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Abstract 

The  Edge  is  a  relative  newcomer  to  organizational  design-one  that  appears  especially  appropriate  for 
contemporary  military  operations,  but  also  raises  issues  regarding  comparative  performance  of  the  Edge 
to  alternate  organizational  designs,  including  more  traditional  hierarchal  configurations.  These  issues 
suggest  that  laboratory  experimentation,  with  coherently  structured  controls  and  manipulations  and  an 
appropriate  data  collection  strategy,  can  offer  substantive  insights  about  the  internal  workings  of  the  Edge 
organization  with  high  levels  of  reliability  and  internal  validity.  Building  upon  prior  command  and  control 
(C2)  research,  we  report  the  preliminary  results  of  our  extension  of  our  campaign  of  computational 
experimentation  to  series  of  laboratory  experiments  using  the  ELICIT  multiplayer  intelligence  game.  Our 
experiments  confirm  the  results  of  earlier  and  companion  computational  experimentation  -  chiefly,  that 
Edge  organizations  outperform  Hierarchy  organizations  in  certain  task  and  environmental  contexts,  and  in 
terms  of  certain  performance  measures.  Our  experiments  also  serve  to  inform  future  computational 
experimentation  through  noting:  1)  Edge  and  Hierarchy  organizations  learn  at  the  same  rate,  but  with 
higher  volatility  within  Edge  configurations;  2)  transforming  from  Hierarchy  to  Edge  configurations  results 
in  either  similar  or  improved  performance  on  subsequent  tasks;  and  3)  transforming  from  Edge  to 
Hierarchy  organizations  results  in  degraded  performance  initially,  with  subsequent  recovery  to  previous 
levels. 


1  In:  Proceedings  International  Command  &  Control  Research  &  Technology  Symposium,  Newport,  Rl 
(June  2007). 
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INTRODUCTION 

The  Edge  [1]  is  a  relative  newcomer  to  organizational  design-one  that  appears  especially  appropriate  for 
contemporary  military  operations,  especially  given  expanding  mission  sets,  such  as  natural  disaster  relief 
[2,3],  pandemic  response  [4,5]  and  stabilization  operations  (see  e.g.,  [6,7]).  The  Edge  proposes  to 
capitalize  upon  fully  connected,  geographically  distributed,  organizational  participants  by  moving 
knowledge  and  power  to  the  edges  of  organizations.  These  changes  in  information  accessibility  and 
organizational  transparency,  coupled  with  distributed  decision  rights  and  empowering  what  would 
traditionally  be  considered  lower-level  organizational  members,  highlight  promising  opportunities  for 
enterprise  efficacy.  However,  the  Edge  organization  also  raises  issues  in  terms  of  comparative 
performance  with  respect  to  alternate  organizational  designs,  including  more  traditional  hierarchal 
configurations.  Such  comparisons  are  of  particular  interest  in  military  and  governmental  contexts  given 
continued  prominence  of  the  paradigmatic,  perhaps  tautological,  concept  of  “unity  of  command,”  [8,9] — 
despite  recognition  that  multinational  coalition  [10],  interagency  [11]  and  interorganizational  networks 
(e.g.,  [12,13])  are  a  common  situational  context  for  defense  officials  and  military  personnel,  particularly 
within  the  more  senior  ranks.  Indeed,  mounting  anecdotal  evidence  suggests  that  a  wide  variety  of  US 
military  organizations — Joint  and  Service — are  testing  new  organizational  forms,  administrative  controls, 
role  assignments  for  senior  officials,  and  communication  protocols,  albeit  without  the  benefit  of  controls 
and  manipulations  inherent  in  more  rigorous  laboratory  experimentation  [14].  US  Strategic  Command,  for 
example,  has  implemented  a  number  of  variants  to  the  traditional  continental  staff  structure  (i.e.,  J-staff) 
typically  found  in  the  Strategic  Apex  [15]  of  combatant  commands,  with  senior  leader  responsibilities 
spanning  multiple  administrative,  functional,  and  cultural  boundaries  simultaneously  [16,17],  Similarly, 

US  Northern  Command  is  actively  pursuing  new  coordination  mechanisms  with  outside  agencies  in  order 
to  execute  its  complex  homeland  security  mission  [12,13].  Senior  defense  officials  have  emphasized 
transformation  -  not  just  in  weapon  systems,  but  also  organizational  processes  -  as  key  to  future  viability 
of  the  US  military.  At  least  anecdotally,  then,  the  Department  of  Defense  is  seeking  to  understand  the 
utility  of  organizational  forms  other  than  traditional  hierarchies. 

Modern  military  organizations  have  adapted  and  evolved  over  many  centuries  (see,  e.g.,  [19-22]). 
Hierarchical  command  and  control  (C2)  organizations  in  particular  have  been  refined  longitudinally  (e.g., 
through  iterative  combat,  training  and  doctrinal  development  [23])  to  become  very  reliable  and  effective  at 
missions  for  which  they  were  designed.  In  contrast,  evidence  to  support  the  putative  benefits  and 
comparative  advantages  proposed  for  Edge  organizations  have  begun  to  emerge  only  very  recently  (e.g., 
see  [24]).  Yet  the  complexity  of  military  practice  [25],  previous  exploration  of  virtual  contexts  [26],  and 
situational  contexts  demanding  high  levels  of  interorganizational  coordination  (e.g.,  [27-30])  seem  to 
indicate  situational  context  in  which  hierarchal  C2  structures  appear  mismatched  or  unsuitable. 

These  issues  suggest  that  laboratory  experimentation,  with  coherently  structured  controls  and 
manipulations  and  an  appropriate  data  collection  strategy,  can  offer  important  insights  about  the  internal 
workings  of  Edge  organizations,  with  high  levels  of  reliability  and  internal  validity.  Building  upon  early 
conceptualizations  of  Edge  organizations  (e.g.,  [1 :20,39]),  our  campaign  of  experimentation  has  focused 
principally  on  the  use  of  computational  models,  which  McKelvey  [31:771]  suggests  as  “constructive 
substitutes”  to  lab  studies  of  organizations.  This  campaign  began  with  a  paper  presented  at  the  2004 
CCRTS  conference  [32],  In  that  paper,  the  relative  advantages  and  disadvantages  of  computational 
experimentation  were  presented,  and  this,  computational  research  method  was  described  in  terms  of  a 
complementary,  empirical  approach.  The  2005  ICCRTS  paper  followed  [33],  in  which  more  than  25 
diverse  organizational  forms  were  compared  and  analyzed,  and  the  Edge  configuration  was  shown  to  be 
theoretically  distinct  and  uniquely  differentiated  from  other  organization  forms  grounded  in  both  theory 
and  practice.  This  2005  paper  also  offered  a  theoretical  discussion  and  set  of  hypotheses  about  the 
performance  of  Edge  and  Hierarchy  organizational  forms  under  different  mission-environmental 
conditions,  and  provided  some  insight  into  relative  characteristics  and  behaviors  of  Hierarchy  and  Edge 
organizations.  The  2006  ICCRTS  paper  [34]  expanded  the  study  to  specify  and  model  four  other,  classic, 
theoretically  grounded  organization  forms:  Simple  Structure,  Professional  Bureaucracy,  Divisionalized 
Form  (i.e.,  M-form,  [35,36]),  and  Adhocracy  [15,37],  We  also  employed  computational  experimentation  to 
compare  and  contrast  empirically  the  relative  performance  of  Hierarchy  and  Edge  organizational  forms, 
using  a  multidimensional  set  of  performance  measures,  under  the  mission-environmental  conditions  at 
two  different  points  in  history:  1)  the  Industrial  Era,  and  2)  the  21st  Century.  Now,  in  a  companion  paper 
[38]  to  this  present  work,  we  are  progressing  systematically  toward  instantiation  and  analysis  of  the  entire 
organization  design  space  (i.e.,  in  a  contingency-theoretic  sense)  of  organizational  forms  and  mission- 
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environmental  contexts.  This  provides  theoretically  grounded,  empirical  results  to  complete  the  kind  of 
“C2  approach  space”  conceptualized  by  Alberts  and  Hayes  [39]. 

The  research  described  in  the  present  paper  extends  the  campaign  of  computational 
experimentation  through  a  series  of  laboratory  experiments  using  the  ELICIT  multiplayer  intelligence 
game  [40].  ELICIT  requires  a  team  of  subjects  performing  the  roles  of  intelligence  analysts  to  collaborate 
— in  a  network-centric,  information-processing  environment — and  identify  a  fictitious  and  stylized  terrorist 
plot.  The  laboratory  setting  enables  the  customary  levels  of  control  and  manipulation  expected  with 
experimentation,  which  provides  for  excellent  reliability  and  internal  validity  of  results  [41].  Additionally,  we 
use  the  results  of  our  prior  computational  experiments  to  suggest  in  part  a  candidate  set  of  research 
hypotheses  for  testing;  to  identify  a  candidate  set  of  dependent  variables  for  measurement;  and  to 
establish  an  empirical  basis  for  further  validation  and  calibration  of  our  computational  models  in  this, 
intelligence-focused,  C2  environment.  Hence  this  part  of  our  campaign  of  experimentation  is  explicitly 
model-driven  (see  McKelvey  [31]),  and  reveals  another,  novel,  powerful  approach  to  C2  research.  Results 
enable  us  to  isolate  some  particularly  powerful  influences  over  and  determinants  of  C2  efficacy — across 
organizational  forms  and  contingency  conditions — and  to  buttress  the  already  solid  foundation  of  external 
validation  that  supports  our  computational  tools.  Results  should  also  help  to  illustrate  the  power  of  the 
ELICIT  game  to  support  compelling  C2  research,  and  to  contribute  important,  new  knowledge  in  terms  of 
both  organization  theory  and  C2  practice. 

In  the  balance  of  the  paper,  we  draw  from  the  organization  studies  literature  and  prior 
computational  experiments  to  motivate  the  set  of  research  hypotheses  examined  through  this  study.  We 
detail  our  research  design,  and  report  in  turn  the  key  findings  and  results.  The  paper  closes  with  a  set  of 
conclusions,  recommendations  for  practice,  and  topics  for  future  research  consistent  with  this  campaign. 

BACKGROUND 

For  more  than  a  half  century,  Contingency  Theory  has  retained  a  central  place  in  organization  studies 
research.  Beginning  with  the  seminal  works  by  Burns  and  Stalker  [42],  Woodward  [43],  and  Lawrence 
and  Lorsch  [44],  organization  theory  has  been  guided  by  the  understanding  that  no  single  approach  to 
organizing  is  best  in  all  circumstances.  Moreover,  myriad  empirical  studies  (e.g.,  [43];  cf.  [45,  46])  have 
confirmed  and  reconfirmed  that  poor  organizational  fit  degrades  performance,  and  many  diverse 
organizational  forms  (e.g.,  Bureaucracy,  see  [69];  M-Form,  see  [35];  Clan,  see  [47];  Network,  see  [48]; 
Platform,  see  [49];  Virtual,  see  [50])  and  configurations  (e.g.,  Machine  Bureaucracy,  Simple  Structure, 
Professional  Bureaucracy,  Divisionalized  Form,  Adhocracy,  see  [15])  have  been  theorized  to  enhance  fit 
across  an  array  of  contingency  factors  (e.g.,  age,  environment,  size,  strategy,  technology). 

In  most  of  this  research,  the  concept  organizational  fit  has  been  treated  in  a  relatively  static 
manner,  with  a  particular  organizational  form  prescribed  to  fit  well  in  a  particular  contingency  context.  For 
instance,  organizational  technology  and  organizational  environment  have  been  studied  extensively  as 
powerful  contingency  factors  (e.g.,  [42,51,52]),  with  alternate  technological  and  environmental 
characteristics  (e.g.,  comprehensibility,  predictability,  complexity,  stability)  related  contingently  with 
different  organizational  forms  (e.g.,  craft,  engineering,  see  [53]).  Indeed,  organization  scholars  have  come 
to  understand  well  how  various  organizational  forms  should  and  do  vary  to  fit  diverse  environmental 
contexts. 

However,  organizational  scholars  (e.g.,  [54-56])  have  noted  widely  that  the  environmental 
contexts  of  many  modern  organizations  are  not  static.  Rather,  organizational  environments  can  change 
rapidly  and  unpredictably,  due  to  multiple  factors  such  as  globalization  [57],  technology  [58,59], 
hypercompetition  [60],  knowledge-based  innovation  [61],  mounting  competition  from  co-evolutionary  firms 
[62],  and  others.  Hence  an  organization  that  achieves  good  fit  with  its  environment  at  one  point  in  time 
may  not  be  able  to  retain  such  fit  longitudinally,  unless  it  changes  structure  in  order  to  maintain  fit — 
dynamically — across  changing  environmental  conditions. 

Indeed,  an  organization  facing  a  constantly  changing  environment  could  fall  into  a  condition  of 
continuous  (disruptive)  change  [63],  or  it  might  take  the  opposite  approach,  striving  instead  toward  a 
single  form  that  is  flexible  and  robust  to  environmental  change  [64].  Alberts  and  Hayes  [1,39]  refer  to  such 
latter  organizational  form  in  terms  of  agility.  In  either  case — and  in  most  cases  in  between — leaders’  and 
policy  makers’  focus  on  static  organizational  fit  is  incommensurate  with  the  dynamics  of  contingent 
organization  demanded  by  disruptive  environmental  change  [65:Ch.  9], 

The  campaign  of  computational  experimentation  summarized  above  has  addressed  this  issue 
directly,  examining  systematically  the  comparative  performance  of  various  organizational  forms  across 
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abruptly  changing  environmental  conditions.  In  essence,  we  are  looking  to  identify  the  best  organizational 
fit  for  abruptly  and  disruptively  changing  environments.  This  campaign  leads  us  to  a  set  of  model-driven 
research  hypotheses  meriting  further  testing  through  experimentation  with  human  subjects.  In  particular, 
Orr  and  Nissen  [34]  conclude  the  following  from  their  computational  experimentation  that  compares  Edge 
and  Hierarchy  organizational  forms.  Each  of  the  hypotheses  summarized  below  derives  from  Alberts  and 
Hayes  [1],  and  most  include  quotations  to  point  the  interested  reader  directly  to  the  original  motivation. 

HO.  Edge  organizations  can  outperform  Hierarchy  organizations  in  demanding  mission 
environmental  contexts. 

Through  computational  experimentation,  the  Edge  organization  is  shown  to  outperform  the  Hierarchy  in 
the  less-familiar,  less-predictable,  more-challenging  environment  referred  to  as  the  “21st  Century  Era.” 
The  previous  authors  conclude  that  the  agility  of  this  Edge  form  enables  it  to  be  more  robust  to 
demanding  mission-environmental  changes.  This  omnibus  hypothesis  is  supported  strongly  by  the  results 
of  the  prior  study. 

HI.  “Power  to  the  Edge  is  the  correct  response  to  the  increased  uncertainty ,  volatility,  and 
complexity  associated  with  [21st  century]  military  operations”  [1:6]. 

Similar  to  the  omnibus  hypothesis  summarized  above,  manipulation  of  the  organizational  environment  in 
computational  experiments  provides  considerable  support  for  this  hypothesis.  The  Edge  organization 
exhibits  considerably  greater  agility,  and  hence  is  more  robust  to  the  challenges  and  demands  of  the  21st 
Century  Era  than  the  Hierarchy  (e.g.,  consider  how  most,  current,  military  C2  is  organized)  is. 

H2.  “The  correct  C2  approach  depends  on  [five]  factors”:  1)  shift  from  static/trench  to 
mobile/maneuver  warfare;  2)  shift  from  cyclic  to  continuous  communications;  3)  volume  and 
quality  of  information;  4)  professional  competence;  and  5)  creativity  and  initiative  [1:19]. 

Similar  to  manipulation  of  the  organizational  environment,  the  previous  authors  demonstrate  that 
improving  the  network  architecture  and  enhancing  professional  competency  increase  organizational 
performance  considerably.  However,  this  result  pertains  to  the  Edge  and  Hierarchy  organizational  forms 
alike.  Hence  improving  network  architecture  and  enhancing  professional  competency  exert  performance¬ 
enhancing  effects  across  organizational  forms,  supporting  elements  2)  and  3)  in  the  hypothesis  stated 
above. 


H3.  “Given  a  robustly  networked  force,  any  one  of  the  six  effective  command  and  control 
philosophies  proven  useful  in  the  Industrial  Era  is  possible”  [1:32]. 

The  network  architecture  manipulation  addresses  this  hypothesis  in  part,  and  the  previous  authors  find 
evidence  that  improving  network  architecture  increases  organizational  agility,  and  makes  the  organization 
more  robust  to  challenges  and  demands  of  the  21st  Century  Era.  However,  their  computational  models  do 
not  represent  each  of  six  different  C2  philosophies  explicitly;  hence  support  for  this  hypothesis  is  limited. 

H4.  People  who  work  together,  over  time,  and  learn  to  operate  in  a  “post  and  smart-pull” 
environment,  will  outperform  similarly  organized  and  capable  people  who  do  not. 

The  professional  competency  manipulation  addresses  this  hypothesis  in  large  part,  but  the  network 
architecture  manipulation  plays  some  role  too  (e.g.,  post  and  smart-pull  environment).  When  focusing  on 
professional  competency  effects,  which  include  people  working  together  over  time,  the  previous  authors 
find  substantial  support  for  this  hypothesis.  A  worthwhile  companion  to  the  experimentation  discussed  in 
this  piece  could  be  exploration  of  alternative  information  dissemination  techniques,  such  as  “smart  push” 
and  others. 

H5.  “The  more  uncertain  and  dynamic  an  adversary  and/or  the  environment  are,  the  more 
valuable  agility  becomes”  [1:124]. 
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Manipulation  of  the  organizational  environment  addresses  this  hypothesis  in  part,  and  results  above  in 
terms  of  comparisons  across  abrupt  environmental  changes  provide  considerable  support  for  this 
hypothesis.  The  Edge  organization  exhibits  considerably  greater  agility,  and  hence  is  more  robust  to  the 
uncertainties  and  dynamics  of  the  21st  Century  Era  than  is  the  Hierarchy. 

H6.  “An  organization’s  power  can  be  increased  without  significant  resource 
expenditures"  [1:1 72]. 

This  hypothesis  is  difficult  to  assess  via  computational  results  developed  by  the  previous  authors,  for  they 
do  not  represent  resource  expenditures  explicitly,  nor  do  they  have  variables  to  measure  organizational 
power.  However,  individual,  agent-level  empowerment  within  an  organization  can  be  operationalized 
using  the  previous  computational  techniques  by  varying  the  level  of  professional  competence  that  an 
agent  holds  relative  to  a  particular  task,  as  well  as  manipulating  the  probability  that  an  agent  seeks 
guidance  from  another  in  the  event  that  an  exception,  or  problem,  is  generated  and  noted.  Indeed,  the 
kinds  of  network  architecture  effects  represented  in  their  model  demand  huge  resource  investments  in 
global  communications  infrastructure.  Such  investments  provide  some  evidence  against  this  hypothesis. 
Alternatively,  the  kinds  of  professional  competency  effects  represented  in  their  model  do  not  demand 
large  resource  investments,  as  simply  changing  policy  to  reduce  job  and  personnel  turnover  can  bring 
about  considerable  improvements  in  knowledge  flows — and  in  turn  organizational  performance. 

To  summarize  the  results  from  computational  experimentation,  empirical  evidence  suggests  that 
the  novel,  poorly  understood,  Edge  organizational  form  outperforms  the  traditional,  well-known  Hierarchy 
across  abrupt  environmental  shifts  (HO).  The  Edge  organization  exhibits  considerably  greater  agility  (HI, 
H5)  than  the  Hierarchy  does,  which  makes  it  more  robust  to  the  challenges  and  demands  of  abrupt 
environmental  change.  Additionally,  improving  network  architecture  and  enhancing  professional 
competency  increase  organizational  performance  considerably,  but  this  result  pertains  to  the  Edge  and 
Hierarchy  organizational  forms  alike  (H2,  H3,  H4).  However,  whereas  network  architecture  enhancements 
demand  huge  resource  investments  in  global  communications  infrastructure,  professional  competency 
improvements  do  not  (H6);  hence  the  latter  may  represent  a  more  prudent  focus  of  attention  and 
resources  than  the  former  does. 

In  terms  of  our  current  experimentation  with  people  in  the  laboratory,  we  focus  principally  upon 
the  omnibus  hypothesis  (HO)  from  above  pertaining  to  organizational  form,  and  we  concentrate  on 
comparison  between  the  Edge  and  Hierarchy.  This  appears  to  represent  the  most  pressing  issue  in  terms 
of  leaders  and  policy  makers,  who  must  organize  enterprises  to  fit  well  in  the  currently  changing 
environment.  This  issue  offers  considerable  theoretical  insight  as  well,  for  the  organization  studies  field 
continues  to  search  for  agile  organizational  forms.  Restating  this  hypothesis  in  terms  appropriate  for 
human  experimentation  in  the  laboratory: 

Hypothesis  1.  People  working  together  in  an  Edge  organization  will  outperform  those  who 
perform  the  same  work  in  a  Hierarchy. 

Additionally,  we  focus  on  the  professional  competency  results  from  above,  for  this  appears  to  be  a  cost- 
effective  approach  to  increasing  the  performance  of  any  organizational  form.  Summarized  simply: 

Hypothesis  2.  Organizations  comprised  of  people  with  greater  professional  competence  will 
outperform  those  with  less-competent  people,  regardless  of  organizational  form. 

Finally,  expanding  upon  the  model-driven  hypotheses  above,  we  draw  from  the  literature  on  knowledge 
management  and  organizational  learning  (e.g.,  see  [95])  to  hypothesize  that  organizations — regardless  of 
form — will  learn  overtime  and  through  task  repetition,  and  that  Edge  organizations  will  learn  more  quickly 
than  Hierarchies: 

Hypothesis  3.  Performance  of  an  Edge  or  Hierarchy  organization  will  increase  over  time  and 
through  task  repetition. 

Hypothesis  4.  Performance  of  an  Edge  organization  will  increase  more  quickly  than  that  of  a 
Hierarchy. 
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RESEARCH  DESIGN 

In  this  section,  we  summarize  the  research  design  used  to  guide  this  series  of  laboratory  experiments. 
Building  directly  upon  the  work  accomplished  by  Parity  [40],  we  employ  the  ELICIT  multiplayer 
intelligence  game  to  examine  how  people  working  together  on  an  information-sharing  and  -processing 
task  perform  across  organizational  configurations.  ELICIT  requires  a  team  of  subjects  performing  the 
roles  of  intelligence  analysts  to  collaborate — in  a  network-centric,  information-processing  environment — 
and  identify  a  fictitious  and  stylized  terrorist  plot.  We  begin  by  describing  this  ELICIT  environment,  and 
then  outline  the  subjects,  protocols,  controls,  manipulations  and  measurements  used  for  experimentation. 

ELICIT  Environment 

The  intelligence  game  involves  a  fictitious  terrorist  plot,  about  which  a  set  of  68  informational  clues  called 
“factoids”  have  been  developed.  Each  factoid  describes  some  aspect  of  the  plot,  but  none  is  sufficient  to 
answer  all  of  the  pertinent  questions  (i.e.,  who,  what,  where,  when).  The  factoids  are  distributed  among 
the  17  players  in  a  series  of  steps:  each  player  receives  two  clues  initially,  followed  by  one  after  five 
minutes  of  play  and  another  after  ten  minutes  have  elapsed.  The  factoid  distribution  is  designed  so  that 
no  single  player  can  solve  the  problem  individually,  and  so  that  the  team  of  players  cannot  solve  the 
problem  until  after  the  final  distribution.  In  other  words,  the  players  must  collaborate  to  solve  the  problem, 
and  they  are  required  to  do  so  for  a  minimum  of  ten  minutes.  Evidence  from  previous  experiments  (e.g., 
[40])  suggests  that  play  requires  substantially  more  time  (e.g.,  an  hour  or  more). 

Subjects  play  the  game  via  client  applications  on  separate,  networked  computer  workstations. 
Each  subject  has  access  to  a  set  of  five  functions  supported  by  the  client:  1 )  List,  2)  Post,  3)  Pull,  4) 
Share,  and  5)  Identify.  The  List  screen  displays  all  factoids  that  a  particular  player  has  received.  For 
instance,  after  the  initial  distribution,  a  player’s  List  screen  would  display  the  two  factoids  distributed  by 
the  server.  Post  enables  a  player  to  have  one  or  more  factoids  displayed  on  a  common  screen  that  can 
be  viewed  by  other  players.  This  represents  one  of  two  mechanisms  for  sharing  information  in  the  game 
(e.g.,  verbal  and  like  communication  is  prohibited  generally  in  most  experiment  protocols).  Pull  represents 
the  complement  to  Post,  as  a  player  can  display  on  his  or  her  List  screen  common  information  that  has 
been  posted.  These  post-pull  functions  are  associated  with  four,  separate  screens,  each  corresponding  to 
the  pertinent  questions  (i.e.,  who,  what,  where,  when)  regarding  the  terrorist  plot;  that  is,  one  screen 
includes  information  regarding  who  (e.g.,  which  terrorist  organization)  might  be  involved,  another  includes 
information  regarding  what  (e.g.,  which  target  might  be  attacked),  and  so  forth  for  information  regarding 
where  and  when  the  attack  might  occur.  Share  represents  the  second  mechanism  for  sharing  information 
in  the  game,  and  enables  players  to  send  factoids  directly  to  one  another.  Finally,  Identify  represents  the 
manner  in  which  subjects  communicate  their  “solutions”  to  the  problem,  indicating  via  the  software  their 
conclusions  regarding  the  pertinent  questions  (i.e.,  who,  what,  where,  when)  regarding  the  terrorist  plot. 

Multiple  versions  of  the  game  have  been  created,  each  of  which  is  structurally  similar  but  distinct. 
For  instance,  each  version  includes  17  players  (and  pseudonyms)  and  a  set  of  68  factoids.  However,  the 
factoids — and  hence  details  of  the  terrorist  plot — are  unique  to  each  version.  Hence  the  potential  exists  to 
play  the  game  multiple  times,  even  with  the  same  group  of  subjects.  Although  time-consuming  and 
tedious,  additional,  structurally  equivalent  versions  of  the  game  can  be  created  as  well.  At  the  present 
time,  four  different  versions  have  been  created  and  shared. 

After  the  game  has  completed,  the  moderator  shuts  down  the  server  application,  and  researchers 
begin  to  analyze  the  transaction  data  captured  by  the  server  in  text-file  logs.  Such  data  include  time 
stamped  entries  for  nearly  every  activity  in  the  networked  ELICIT  environment,  including,  for  instance, 
when  and  which  factoids  are  distributed  to  each  player,  when  and  which  factoids  are  posted  to  which 
common  screens,  when  and  which  common  screens  are  viewed  by  each  player,  when  and  which  factoids 
are  shared  between  each  player,  and  the  time  stamped  results  of  each  player’s  Identify  attempt.  The 
game  requires  considerable  cognitive  and  collaborative  effort  to  play  well  (i.e.,  identify  the  pertinent 
details  of  a  terrorist  plot),  but  such  effort  is  within  the  capabilities  of  many  people  and  groups. 

Subjects 

Subjects  in  this  experiment  represent  a  combination  of  (mostly)  masters  and  PhD  students  and  (a  few) 
faculty  members  in  at  a  major  US  university.  Subjects  are  grouped  into  four  sections:  1)  Group  A  is 
comprised  principally  of  PhD  students  in  information  science;  2)  Group  B  is  comprised  principally  of 
masters  students  enrolled  in  an  advanced  C2  course;  3)  Group  C  is  comprised  principally  of  masters 
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students  enrolled  in  an  introductory  C2  course;  and  4)  Group  D  is  comprised  principally  of  masters 
students  with  a  special  operations  and/or  intelligence  background. 

A  total  of  68  subjects  range  in  age  from  22  to  62  years  =  35.8,  o  =  8.52),  and  possess  between 
1  and  38  years  of  work  experience  (/v  =  11 .82,  o  =  8.41 ).  All  subjects  have  undergraduate  college 
degrees,  and  42%  have  graduate  degrees.  Hence  the  subjects  are  representative  in  part  of  the  kinds  of 
relatively  experienced  and  well-educated  people  who  serve  as  professional  intelligence  analysts, 
particularly  in  national  intelligence  agencies.  Further,  all  of  the  subjects  have  direct  military  or  government 
service,  and  some  have  worked  professionally  in  military  or  government  intelligence  organizations.  Hence 
the  subjects  are  also  representative  military  and  government  employees  who  serve  as  professional 
intelligence  analysts.  This  representative  sample  serves  to  enhance  the  external  validity  of  the  study. 

However,  none  of  the  subjects  works  currently  as  a  professional  intelligence  analyst,  and  none  of 
the  three  groups  of  subjects  has  worked  together  previously  in  an  intelligence  capacity.  In  this  regard,  the 
laboratory  introduces  some  artificiality  into  the  experiment.  Additionally,  despite  the  considerable  level  of 
realism  designed  into  the  ELICIT  game,  the  information-sharing  and  -processing  task  is  limited 
intentionally,  so  that  people  can  play  the  game  within  an  hour  or  two,  and  the  networked-computer, 
ELICIT-mediated  task  environment  does  not  enable  all  of  the  same  kinds  of  media-rich  communication 
modalities  (e.g.,  telephone,  video  teleconference,  face-to-face  interpersonal  and  group  interaction)  likely 
to  be  found  in  operational  intelligence  organizations  in  the  field.  These  factors  serve  to  limit  the  external 
validity  of  the  study.  Limitations  such  as  these  are  inherent  within  laboratory  experimentation  [32],  and 
call  for  the  use  of  other,  complementary  research  methods  (e.g.,  fieldwork,  see  [66]). 

Protocols 

Subjects  are  pre-assigned  to  play  specific  roles  (e.g.,  as  identified  via  pseudonyms)  in  the  game,  and  to 
the  extent  possible,  each  subject  plays  the  same  role  in  every  experiment  session.  In  this  particular 
experiment,  subjects  are  pre-assigned  to  roles  based  upon  their  level  of  work  experience.  This  is  similar 
to  the  manner  in  which  professional  analysts  are  assigned  to  specific  roles  in  operational  intelligence 
organizations  in  the  field,  and  hence  helps  to  ground  this  experiment  though  conformance  to  practice. 

This  approach  contrasts  a  bit  with  that  of  randomized  assignment  imposed  in  some  prior  studies  (cf.  [40]), 
emphasizing  our  concern  for  realism  over  replication. 

Subjects  read  about  the  experiment,  and  consent  formally  to  participate  in  it.  When  all  ELICIT 
clients  have  connected  with  the  server,  subjects  sit  down  at  the  appropriate  workstations,  are  informed 
verbally  about  the  nature  of  the  experiment,  and  are  asked  to  read  a  set  of  instructions  pertaining  to  both 
the  experiment  and  the  ELICIT  environment.  The  instructions  for  Group  A  subjects  are  included  in 
Appendix  A  for  reference.  Subjects  are  encouraged  to  ask  questions  throughout  this  process.  When 
subjects  have  read  the  instructions,  and  have  had  their  questions  answered  satisfactorily,  they  indicate 
via  the  ELICIT  client  that  they  are  ready  to  begin. 

In  this  particular  experiment,  each  of  the  four  subject  groups  participates  separately  (e.g.,  on  a 
different  day  of  the  week),  and  each  group  participates  in  a  total  of  four  experiment  sessions,  each  time 
playing  a  different  version  of  the  game  (i.e.,  Versions  1  -  4).  For  Groups  A,  B,  and  C,  each  of  the  four 
experiment  sessions  is  spaced  roughly  one  week  apart;  for  Group  D,  experiment  sessions  are  conducted 
twice  a  week  for  two  weeks.  This  provides  time  for  subjects  to  reflect  upon  the  game,  and  to  interact  with 
one  another  outside  of  the  laboratory  (e.g.,  as  collaborating  professional  intelligence  analysts  do),  but 
given  that  the  subjects  have  many  responsibilities  outside  of  the  laboratory  experiments,  this  provides 
time  also  for  subjects  to  forget  about  specific  aspects  of  each  session  (e.g.,  as  multitasking  professional 
intelligence  analysts  do).  Hence  some  learning  and  forgetting  outside  of  the  laboratory  environment  takes 
place  between  experiment  sessions.  The  specific  schedule  of  play  is  described  below. 

Subjects  are  instructed  not  to  reveal  their  pseudonyms  to  one  another  during  the  game.  Indeed, 
they  are  instructed  not  to  talk  or  communicate  with  one  another  during  the  game  via  any  mechanism 
outside  of  the  two  summarized  above  (i.e.,  post-pull,  share).  This  simulates  the  kind  of  globally 
distributed,  network-centric  environment  in  which  much  intelligence  work  takes  place  operationally  today. 
Additionally,  subjects  are  allowed  to  send  handwritten  “postcards”  directly  to  one  another  at  periodic 
intervals  in  two  of  the  four  groups.  Postcards  contain  the  same  information  associated  with  an  Identify 
function  (i.e.,  who,  what,  where  and  when  details).  This  enriches  the  communication  media  available  to 
the  subjects  beyond  the  artificially  limiting  factoid  distribution  enabled  by  the  ELICIT  software.  To 
preserve  anonymity,  subjects  send  such  postcards  via  the  Experiment  Moderator,  who  shuffles  and 
delivers  them  to  their  intended  recipients.  Hence  the  sender  of  a  postcard  knows  only  the  pseudonym  of 
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the  receiver,  and  vice  versa.  In  addition  to  enriching  the  communication  media,  such  postcards  also 
capture  in  part  the  mental  models  [96-101]  of  subjects  at  various  points  of  game  play.  A  total  of  one 
postcard  is  allowed  at  each  interval,  with  four  or  five  intervals,  coinciding  approximately  with  the  15-,  25-, 
35-,  45-  and  55-minute  marks  in  the  game.  Mental  models,  captured  on  the  experimental  postcards,  are 
collected  from  each  player,  regardless  of  group,  by  the  Experiment  Moderator.  However,  as  described  in 
the  manipulations  section  below,  in  only  two  of  the  four  groups  are  the  “postcards”  delivered  to  their 
intended  recipients. 

Subjects  are  given  incentives  to  play  the  game  well,  as  participation  and  performance  are 
factored  into  the  evaluation  of  students’  coursework.  Subjects  are  given  incentives  also  for  personal  gain 
(e.g.,  a  “point”  is  awarded  for  an  individual  person  that  identifies  the  plot  correctly  in  the  shortest  period  of 
time)  as  well  as  for  group  gain  (e.g.,  a  “point”  is  awarded  for  all  members  of  the  team  that  identify  the  plot 
correctly  in  the  shortest  period  of  time).  This  is  intended  to  mimic  the  dual  nature  of  incentives  that  exist  in 
professional  intelligence  environments,  where  people  must  cooperate  for  the  organization  to  perform  well, 
but  who  also  compete  against  one  another  for  limited  rewards  such  as  wage  increases,  promotions, 
desirable  job  assignments,  and  like  intrinsic  and  extrinsic  factors.  The  incentive  structure  is  thus 
somewhat  analogous  to  the  profit-sharing  incentive  system  described  by  Groves  [67],  Further,  of  the  44 
team-individual  reward  strategies  identified  by  Cacioppe  [68],  the  game  incentive  structure  provided 
public  recognition  (RIO),  praise  (R11),  feedback  (R12),  team-building  (R19)  and  team  attention  (R20). 
Cacioppe  [68]  describes  these  reward  and  recognition  strategies  as  falling  between  extrinsic  and  intrinsic 
rewards,  and  specifically  ascribes  their  utility  for  the  two  phases  of  the  team  life  cycle  most  critical  to  the 
experiment — establishing  itself  (stage  2)  and  performing  the  task  (stage  3). 

Each  subject  is  instructed  to  use  the  Identify  function  only  once  during  game  play.  This 
represents  the  manner  in  which  formal  conclusions  about  terrorist  plots  in  practice  are  taken  very 
seriously,  and  how  they  impact  other  organizations  (e.g.,  an  operational  organization  may  declare  a  state 
of  emergency  in  preparation  for  or  response  to  a  suspected  terrorist  plot).  Hence  each  player  in  the  game 
is  expected  to  wait  until  he  or  she  is  relatively  confident  about  the  plot  before  sending  an  “official”  notice. 
Alternatively,  the  use  of  postcards  above  allows  subjects  to  exchange  the  same  information  informally 
with  select  other  players.  This  represents  the  manner  in  which  informal  hypotheses  are  discussed  and 
compared  frequently  within  operational  intelligence  organizations. 

The  game  can  end  in  either  of  two  ways:  1)  when  all  players  make  their  identification,  or  2)  when 
the  Moderator  must  end  the  game  due  to  time  constraints.  Generally,  subjects  are  not  told  the  results  of 
the  game  (e.g.,  plot  details)  until  after  all  four  versions  of  the  game  have  been  played.  This  represents  in 
part  the  kind  of  equivocality  inherent  in  intelligence  work:  analysts  are  rarely  certain  about  any  suspected 
plot,  and  many  are  required  to  work  on  multiple  plots  either  simultaneously  or  sequentially.  Again,  we  go 
to  considerable  lengths  to  enhance  the  realism  of  the  game — and  hence  external  validity  of  the  results. 

Finally,  multiple  instruments  are  administered  to  the  subjects,  at  various  points  in  time  during  the 
series  of  experiments.  None  of  these  instruments  is  administered  during  game  play.  They  are  described 
in  the  measurements  section  below. 

Controls 

As  noted  above,  each  subject  is  pre-assigned  a  specific  role  to  play,  and  is  intended  to  play  such  specific 
role  through  each  version  of  the  game.  Each  version  of  the  game  is  structurally  equivalent,  and  both  the 
ELICIT  software  and  physical  laboratory  environments  are  invariant  across  experiment  sessions.  Further, 
via  the  instruments  administered  to  the  subjects  and  enacted  within  the  ELICIT  environment,  researchers 
have  the  ability  to  control  for  myriad  factors  (e.g.,  personality,  information-sharing,  experience)  ex-post  to 
the  experiment  sessions.  In  general,  we  strive  to  control  every  aspect  of  the  environment  and  experiment 
that  is  not  manipulated  expressly  as  described  below.  To  the  extent  possible,  we  also  match  teams  for 
gender,  Service,  military  rank,  and  age,  achieving  the  greatest  uniformity  between  Groups  B  and  C. 

To  replicate  real-world  organizations,  more  experienced  personnel  are  given  roles  of  greater 
perceived  responsibility  -  i.e.,  cross-team  coordinator,  team  leader  -  in  the  Hierarchy  configuration,  as 
operationalized  in  the  pseudonyms  and  role  assigned  to  each  player  in  the  ELICIT  environment.  This 
preference  for  assigning  more  experienced  personnel  is  continued  into  the  Edge  configurations  to  provide 
consistency  across  all  four  groups  and  all  16  experimental  sessions.  In  the  case  of  an  absent  player  in 
one  of  these  five  key  positions  in  the  Hierarchy  configuration,  an  experienced  subject  was  promoted  to  fill 
the  vacancy  in  a  style  similar  to  real-world  organizations.  A  team  leader,  for  example,  would  serve  as  the 
cross-team  coordinator  in  his  or  her  absence;  similarly,  the  most  experienced  team  member  would  serve 
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for  the  team  leader  in  his  or  her  absence.  Less  experienced  subjects  and  those  with  known  absences 
during  the  experimental  period  are  assigned  to  team  member  positions  under  both  configurations  in  order 
to  minimize  the  impact  of  missing  or  transitory  players  on  the  experimental  design.  If  the  group  played 
with  fewer  than  17  subjects,  the  experiment  moderator  would  ensure  that  the  missing  player’s  factoids  (4 
in  total)  are  available  to  the  other  players  via  the  software. 

On  occasion,  the  same  subject  plays  with  two  different  experimental  groups.  To  minimize  the 
effect  of  this  play,  the  factoid  sets  are  manipulated  such  that  they  are  structurally  equivalent,  but 
cosmetically  unique  (e.g.,  Blue  Group  ->  Green  Group),  during  each  round.  Any  errors  or  misspellings 
noted  in  the  factoid  sets  are  repeated  in  the  homomorphic  substitutes  to  ensure  one-to-one 
correspondence  of  all  factoid  sets  used  during  a  particular  round  of  experimentation.  Repeat  subjects  are 
assigned  positions  of  lowest  relative  responsibility  in  the  Hierarchy  configuration,  as  well  as  a  new 
pseudonym  (which  ensures  a  different  distribution  of  factoids)  during  repeat  play. 

Manipulations 

The  manipulations  center  on  the  research  hypotheses  motivated  and  summarized  above.  To  test  the  first 
hypothesis  about  comparative  performance  of  Edge  and  Hierarchy  forms,  subjects  are  assigned  to 
corresponding  experimental  environments.  No  specific  manipulations  are  associated  with  the  second 
hypothesis.  Rather,  we  intend  to  gauge  professional  competence  of  the  subjects  outside  of  the  game 
play,  and  to  seek  correlation  and  understanding  via  ex-post  analysis.  The  two  hypotheses  regarding 
learning  are  addressed  via  the  Hierarchy-Edge  manipulation  described  above. 

Hierarchy 

In  the  Hierarchy  organization  manipulation,  subjects  are  assigned  to  play  roles  within  a  three-level, 
functional,  hierarchical  organization  as  depicted  in  Figure  1.  An  overall  leader  (i.e.,  labeled  “1”)  is 
responsible  for  the  intelligence  organization  as  a  whole,  and  has  four  functional  leaders  (i.e.,  labeled  “2,” 
“6,”  “10,”  “14”)  reporting  directly.  Each  such  leader  in  turn  has  three  analysts  (e.g.,  labeled  “3,”  “4,”  “5”) 
reporting  directly,  and  is  responsible  for  one  set  of  details  associated  with  the  terrorist  plot.  For  instance, 
Subleader  2  and  team  would  be  responsible  for  the  “who”  details  (e.g.,  which  terrorist  organization  is 
involved)  of  the  plot,  Subleader  6  and  team  would  be  responsible  for  the  “what”  details  (e.g.,  what  the 
likely  target  is),  and  so  forth  for  “when”  and  “where.”  Subjects  are  shown  this  organization  chart,  told  of 
their  responsibilities  within  the  organization,  and  provided  with  a  short  description  of  the  hierarchy. 

Additionally,  the  ELICIT  software  limits  subjects’  Post  and  Pull  access  to  specific  common 
screens  within  this  manipulation.  Specifically,  those  players  in  the  “who”  group,  for  instance,  are  allowed 
to  Post  to  and  Pull  from  only  one  of  the  four  common  screens  (i.e.,  the  “who”  screen)  noted  above. 
Comparable  restrictions  apply  to  players  in  the  other  three  functional  groups.  The  only  exception  applies 
to  the  Leader  1 ,  who  has  post-pull  access  to  all  four  common  screens.  Further,  we  limit  postcards  to 
immediate  superiors  and  subordinates  within  the  organization.  These  manipulations  reinforce  the 
functional  and  hierarchical  nature  of  the  Hierarchy  organizational  form  represented. 


Who  Group  What  Group  Where  Group  When  Group 


Figure  1:  Hierarchy  Organization 

Alternatively,  players  are  allowed  to  use  the  Share  function  to  send  factoids  to  any  of  the  16  other 
players  in  the  entire  organization.  This  serves  to  capture  the  “flattening”  effect  of  e-mail  and  like,  now- 
ubiquitous  communication  modes  that  enable  peer-to-peer  collaboration  across  formal  organizational 
boundaries.  We  note,  however,  that  such  Share  function  is  limited  to  sharing  factoids  only:  no  free-form  or 
other  information  can  be  exchanged  in  this  direct  manner. 
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In  terms  of  the  game  ending,  in  this  manipulation,  the  game  ends  when  all  the  players  identify  the 
plot  details,  or  when  the  game  times  out.  However,  the  incentive  structure  ensures  such  that  players  other 
than  the  leader  receive  individual  recognition  if  and  only  if  his  or  her  pre-selected  leader  identifies  the  plot 
correctly  and  in  less  time  than  the  other  two  teams.  This  represents  the  manner  in  which  leaders  of  many 
hierarchical  organizations  speak  for  the  organization  as  a  whole,  and  it  captures  the  important 
information-sharing  task  of  ensuring  that  such  leader  is  informed  well. 

Edge 

In  the  Edge  organization  manipulation,  there  are  no  pre-assigned  leaders  or  functional  groups  established 
in  advance  of  the  experiment.  Rather,  consistent  with  current  Edge  conceptualizations,  the  group  is 
leaderless  and  without  form — what  Mintzberg  terms  Adhocracy.  As  noted  above,  the  players  are  pre¬ 
assigned  to  specific  roles  (i.e.,  pseudonyms)  within  the  game,  but  the  various  roles  reflect  no  hierarchical 
or  functional  differences  from  one  another.  As  with  the  hierarchy  manipulation  above,  subjects  are  told 
about  this  organizational  arrangement,  and  are  provided  with  a  short  description  of  the  Edge  as  an 
organizational  form.  We  characterize  the  nature  of  this  Edge  manipulation  in  Figure  2. 
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Figure  2  Edge  Organization 


Without  an  overall  leader  or  functional  groups,  subjects  must  decide  for  themselves  who  works  on 
which  aspects  of  the  problem,  and  who  posts,  pulls  and  exchanges  information  with  whom.  With  this 
manipulation,  the  ELICIT  software  does  not  limit  subjects’  Post  and  Pull  access  to  specific  common 
screens;  that  is,  in  contrast  to  the  hierarchy  manipulation  above,  any  player  can  post  to  and  pull  from  any 
of  the  four  common  screens  (i.e.,  “who,”  “what,”  “when,”  “where”).  In  further  contrast,  any  player  can  send 
a  postcard  to  any  other  player,  albeit  within  the  same  format,  frequency  and  number  constraints 
established  for  the  hierarchy  manipulation.  Consistent  with  the  other  manipulation  is  the  Share  function, 
through  which  any  player  can  share  factoids  directly  with  any  other. 

In  terms  of  the  game  ending,  in  this  manipulation,  the  game  ends  when  all  players  Identify  the 
plot  details,  or  when  the  game  times  out.  This  represents  the  manner  in  which  flat,  leaderless 
organizations  require  some  consensual  decision  making,  and  it  captures  the  important  information¬ 
sharing  task  of  ensuring  that  all  participants  are  informed  well.  To  ensure  comparability  with  the  hierarchy 
results,  however,  after  the  game  has  completed,  participants  are  asked  to  elect  an  emergent  leader,  and 
this  subject’s  game  performance  (e.g.,  evidenced  via  the  Identify  function)  is  used  for  comparison  with 
that  of  the  leader  (i.e.,  Leader  1)  in  the  hierarchy  manipulation. 

Manipulation  Sequence 

Each  of  the  three  subject  groups  is  assigned  to  a  unique  manipulation  sequence  as  summarized  in  Table 
1,  and  each  group  plays  all  four  versions  of  the  game  once  (i.e.,  each  group  plays  a  total  of  four  times). 
Group  A  plays  according  to  the  Edge  manipulation  all  four  times.  Because  we  know  relatively  little  about 
Edge  organizations — particularly  how  they  form  and  learn  overtime — this  manipulation  provides 
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longitudinal  data  for  exploration.  Groups  B  and  C  play  twice  each  in  the  Hierarchy  and  Edge 
manipulations,  but  the  order  of  play  is  reversed.  This  reduces  potential  confounding  from  learning  effects 
associated  with  order  of  play.  These  groups  also  play  twice  within  each  manipulation  (e.g.,  twice  in 
Hierarchy,  then  twice  in  Edge)  before  reversing.  This  allows  two  experiment  sessions  for  learning  within  a 
particular  organizational  form  to  occur.  The  contrast  between  Group  B  and  Group  C  performance  reveals 
between-subjects  effects;  the  contrast  between  Hierarchy  and  Edge  manipulations  reveals  both  within- 
and  between-subjects  effects;  and  both  individual  and  organizational  learning  over  time  reveals  within- 
subjects  effects.  Group  D  plays  according  to  the  Hierarchy  manipulation  all  four  times,  offering  a  contrast 
to  Group  A.  As  a  note,  the  individual  player  represents  the  primary  unit  of  analysis,  but  both  individual 
and  organizational  levels  of  analysis  are  considered  in  this  experiment.  A  second  manipulation  involves 
allowing  or  disallowing  each  subject  to  share  his  or  her  evolving  mental  model  of  the  impending  terrorist 
attack  via  a  highly  structured  data  collection  mechanism  and  timing  criterion — operationalized  as  the 
sharing  or  withholding  of  “postcards”  as  outlined  above.  Further  details  about  this  manipulation  are 
available  from  the  authors,  and  results  from  this  manipulation  are  reserved  primarily  for  discussion  in 
future  work,  with  some  limited  explication  offered  in  this  paper. 


Group 

Session  1 
V4 

Session  2 
V3 

Session  3 
V2 

Session  4 
VI 

A -PhD 

E  -  no  MM 

E  -  no  MM 

E  -  no  MM 

E  -  no  MM 

B  -  Advanced  C2 

H-MM 

H-MM 

E-MM 

E-MM 

C  -  Introductory  C2 

E-MM 

E-MM 

H-MM 

H-MM 

D  -  SOF  /  Intel 

H-no  MM 

H-no  MM 

H-no  MM 

H-no  MM 

Key: 

VI -V4:  Elicit  Version  1-4 
H:  Hierarchy  manipulation 
E:  Edge  manipulation 

MM:  Mental  models  distributed  between  players 
No  MM:  Mental  models  not  distributed  between  players 

Table  1:  Manipulation  Sequence 


Measurements 

This  section  describes  our  operationalization  schema  for  performance,  professional  competence,  and 
organizational  learning. 

Performance.  The  first  hypothesis  addresses  comparative  organizational  performance  of 
Hierarchy  and  Edge  organizations,  and  the  performance  of  individuals  within  Hierarchy  and  Edge 
organizations.  In  this  experiment,  performance  is  operationalized  as  a  two-dimensional  dependent 
variable  comprised  of:  1 )  time  to  identify  plot  details  correctly;  and  2)  accuracy  of  the  plot  identification. 
This  measurement  construct  is  informed  by  our  computational  experiments  (e.g.,  see  [33,  70-72]),  in 
which  time  and  accuracy  (related  to  risk)  reveal  consistently  insightful  results.  The  measurement 
construct  is  also  informed  by  literature  in  the  psychological  and  organizational  domains  that  suggest  a 
trade-off  exists  between  time  and  accuracy  in  tasks  requiring  high  cognition  and/or  advanced  motor  skills 
(e.g.,  see  [73-78])  at  both  the  individual  and  team/group  levels  of  analysis. 

In  the  first  component,  time  pertains  to  when  a  subject  submits  his  or  her  identification  of  the 
terrorist  plot,  with  group  performance  with  respect  to  time  operationalized  as  the  mean  submission  time  of 
all  subjects  participating  during  the  experimental  session.  For  ease  of  comparison,  the  scales  for  both 
measurements  are  normalized  to  a  0-1  scale,  with  1  being  more  desirable  in  both  cases  (e.g.,  faster, 
more  accurate).  Measuring  and  normalizing  time  is  straightforward,  as  the  time  for  each  subject’s 
identification  is  logged  to  the  nearest  second  by  the  software.  To  ensure  that  the  measurements  are 
meaningful  when  compared  against  values  from  all  16  experimental  sessions,  we  determine  that 
identifications  made  at  the  same  ‘clock’  time  during  two  different  sessions  (e.g.,  after  2200  seconds  has 
elapsed  since  the  start  of  Session  1  and  after  2200  seconds  has  elapsed  since  the  start  of  Session  2)  are 
to  be  considered  exactly  equal.  Each  subject’s  identification  time  is  thus  calculated  using  Equation  1: 

_  3  896  -  identification  _  time  ^q  ^  ^ 

”  3896 

In  Equation  1, 3896  represents  the  maximum  time  elapsed  (in  seconds)  during  all  16  experiments.  An 
example  in  which  any  subject  identifies  at  2200  seconds  after  any  experimental  session  begins  becomes: 
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3896-2200 

3896 


=  0.44 


In  the  second  component  of  performance,  accuracy  refers  to  when  the  subject  has  identified  the 
specific  details  of  an  impending  terrorist  attack  -  i.e.,  who,  what,  where  and  when,  with  group 
performance  for  accuracy  operationalized  again  as  the  mean  accuracy  of  identifications  provided  by 
subjects  during  the  experimental  session.  Sufficient  information  is  contained  within  the  factoid  sets  such 
that  the  subjects  can  discern  the  group  responsible  for  the  attack  (“who”),  the  target  of  the  attack  (“what”), 
the  country  in  which  the  attack  will  take  place  (“where”)  and  the  month,  date  and  time  of  the  terrorist 
attack  (“when”).  For  the  results  reported  here,  we  operationalize  accuracy  according  to  strict  criteria,  with 
a  subject  receiving  a  high  score  on  accuracy  if  his  or  her  identification  of  the  terrorist  attack  reduced 
decisionmaker  uncertainty  exactly.  Our  model  is  thus  consistent  with  Heuer’s  [79,80]  description  of 
intelligence  analysts  as  agents  that  filter  and  make  sense  of  scattered  and  potentially  incomplete 
information  on  behalf  of  policy  makers,  informing  policy  makers  by  reducing  uncertainty  about  complex 
topics.  A  subject  receives  credit  for  his  or  her  identification  under  this  strict  schema  if  it  matches  the 
correct  response  exactly,  with  some  reasonable  exceptions  for  natural  language  equivalents  and  use  of 
“military  time”  (i.e.,  24-hour  clock)  by  many  respondents.  A  point  is  awarded  for  each  component  of  the 
correct  answer  -  group,  target,  country,  month,  date,  and  time  of  day  -  and  then  normalized  to  a  scale 
from  0  to  1 ,  with  equal  weighting  for  the  who,  what,  where,  and  when  components.  No  points  are  awarded 
for  blank  (i.e.,  non-)  answers.  An  illustration  of  the  operationalization  is  provided  at  Appendix  E. 

Professional  competence.  The  second  hypothesis  addresses  professional  competence.  We 
capture  demographic  and  experiential  information  (e.g.,  education,  work  experience,  intelligence 
experience)  from  subjects  in  order  to  pre-assign  them  to  roles  in  the  ELICIT  game.  Such  demographic 
information  is  used  to  correlate  and  understand  the  role  and  distribution  of  professional  competence  on 
individual  and  organizational  performance.  This  knowledge  inventory  instrument  is  included  in  Appendix  B 
for  reference.  Measures  used  in  this  section  focus  on  the  subjects’  work  experience  (years), 
military/government  experience  (years),  and  intelligence  experience  (years). 

Learning.  The  third  and  fourth  hypotheses  address  learning.  We  operationalize  learning  as  the 
change  in  performance  overtime  and  repetition,  and  use  the  same,  two-dimensional  dependent  variable 
summarized  above.  Specifically,  we  measure  the  change  in  performance  across  the  four  experiment 
sessions — blocking  by  organizational  form. 


RESULTS 

We  discuss  the  results  of  our  laboratory  experiments  in  the  order  outlined  above:  performance, 
professional  competence,  and  learning. 


Consistency  of  Observations  with  Proposed  Design 

We  first  check  that  our  experimentation  follows  our  research  design.  Basic  correlations  reveal  that  our 
primary  manipulations-organizational  form  and  mental  models-are  not  highly  correlated,  a  condition 
important  to  simplifying  the  interpretation  of  results  from  multivariate  research  designs  [81 :775],  Given 
the  design  as  described  in  Table  1 ,  the  Spearman’s  Rho  coefficients  for  our  independent  variables  are  as 
expected  (Appendix  D)  and  confirm  the  robustness  of  our  basic  factorial  design  for  the  stated  hypotheses. 

Organizational  Form  and  Mental  Models 

We  conduct  experimentation  during  16  sessions  over  a  36  day  period.  Groups  occasionally  play  ELICIT 
with  fewer  than  the  requisite  17  players,  but  our  experimental  protocol  ensures  that  in  all  but  one  case, 
subjects  can  access  all  information  needed  to  completely  discern  all  details  about  the  impending  terrorist 
attack.2  A  total  of  68  unique  subjects  play  the  game  from  1  to  8  times  (p  =  3.51 ,  o  =  1 .71 ),  with  90%  of 
subjects  submitting  an  identification  during  game  play  and  thus  providing  210  cases  for  evaluation  at  the 
individual  level  of  analysis.  The  authors  play  ELICIT  with  Group  A  (PhD  Group),  however,  the  analysis 
omits  any  associated  data.  Subjects  occasionally  submit  their  identification  of  the  terrorist  attack  more 
than  once,  but  for  consistency  in  the  analysis  and  with  the  instruction  set  to  the  players,  all  results  reflect 
only  the  subject’s  first  identification,  regardless  if  a  subsequent  identification  was  more  accurate.  The 


2  The  exceptional  case  represents  the  pilot  experiment  in  which  the  subjects  do  not  receive  a  factoid  related  to  the 
exact  hour  of  the  impending  attack.  All  other  information,  however,  is  available  to  the  players  during  this  session. 
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distribution  of  these  210  cases  between  and  within  the  two  manipulations  is  provided  in  Table  2  below. 
More  detailed  descriptive  statistics  are  provided  in  Appendix  D  (Table  4). 


Organizational  Form 

Total 

Edge 

Hierarchy 

Mental  Exchanged 

66 

62 

128 

Model  Not  Exchanged 

46 

36 

82 

Total 

112 

98 

210 

Table  2:  Cross-tabulation  of  Observations 

The  two  components  of  the  dependent  variable  performance  (i.e.,  time  and  accuracy)  are  checked  for 
normality  using  probability  plots,  with  acceptable  results,  and  interestingly,  are  not  correlated  (r  =  -0.058, 
p>0.40).  Normal  distribution  of  and  lack  of  covariance  between  the  dependent  variables  supports  the 
appropriateness  of  utilizing  ANOVA  to  determine  if  the  manipulations  resulted  in  statistically  significant 
differences  among  the  sample  populations.  Coupled  with  the  low  correlation  between  the  primary 
manipulations  (0.044,  p>0.52),  the  normality  of  distribution  also  supports  use  of  MANOVA  techniques  for 
assessing  our  results  [81]. 

Within  and  between  the  two  manipulations,  the  means  of  the  two  performance  factors  are 
different,  and  for  half  of  the  groups,  the  difference  of  the  means  appears  significant.  For  example,  when 
mental  models  are  exchanged,  players  in  the  Edge  configurations,  on  average,  score  0.47  on  the  time  of 
their  identifications  (recall  that  0  would  represent  the  slowest  time  for  any  player  to  identify  the  details  of 
the  terrorist  attack  in  any  experimental  round,  while  1  would  represent  the  fastest  time  for  any  player  to 
identify  details  about  the  terrorist  attack  in  any  experimental  round).  Under  the  same  condition,  however, 
players  in  Hierarchy  configurations,  on  average,  score  0.25  for  time  of  their  identifications,  or  0.47  -  0.25 
=  0.22  slower  than  Edge  groups  on  our  normalized  scale.3 

Similarly,  players  in  the  Hierarchy  configurations,  when  allowed  to  exchange  their  mental  models 
of  the  terrorist  attack  (even  in  the  very  limited,  structured  manner  provided  by  the  experimental  protocol), 
score,  on  average,  0.68  for  accuracy  (recall  that  0  would  represent  a  completely  inaccurate  identification, 
while  1  would  represent  a  completely  accurate  identification).  Players  in  Hierarchy  configurations,  when 
not  allowed  to  exchange  mental  models,  score,  on  average,  0.56  for  accuracy  -  a  difference  of  0.68  - 
0.56  =  0.12  slower  than  with  the  exchange  of  mental  models  on  our  normalized,  dimensionless  scale. 

One  reasonable  interpretation  of  these  results  is  that  the  exchange  of  knowledge  (operationalized  as 
“postcards”)  among  the  players  slows  the  hierarchy,  but  yields  greater  accuracy  over  the  case  of  simply 
exchanging  information  (operationalized  as  “factoids”).  The  Edge,  however,  demonstrates  no 
degradation  in  time  when  mental  models  are  exchanged,  with  a  nearly  similar  improvement  in  accuracy. 
The  Edge,  it  appears  at  first  glance,  can  absorb  a  higher-level  cognitive  task  (i.e.,  exchange  of 
interpretation  vs.  exchange  of  existing  data)  more  easily  than  the  Hierarchy,  although  both  forms  benefit 
in  terms  of  improved  accuracy. 

Organizational  Form 

Hypothesis  1  suggests  that  people  working  together  in  an  Edge  organization  will  outperform  those  who 
perform  the  same  work  in  a  Hierarchy.  To  test  this  hypothesis,  we  use  univariate  ANOVA  to  determine 
whether  the  differences  in  the  means  of  individuals  working  in  an  Edge  and  individuals  working  in  a 
Hierarchy  are  significant.  For  example,  the  mean  time  for  players  working  within  Edge  configurations, 
regardless  of  the  exchange  of  mental  models,  is  0.47.  This  result  appears  to  contrast  with  the  mean  time 
of  0.33  for  players  working  with  Hierarchy  configurations  (regardless  of  the  exchange  of  mental  models), 
suggesting  that  persons  in  a  Hierarchy  configuration  identified  the  terrorist  attack  details  more  slowly  than 
their  Edge  counterparts.  Similarly,  the  mean  accuracy  for  players  working  within  Edge  configurations  is 
0.69,  while  the  mean  accuracy  for  players  working  within  Hierarchy  configurations  is  0.66,  a  lesser 


3  To  convert  from  the  normalized  scale  to  seconds,  one  would  multiply  by  3869  seconds.  Thus  the  difference 
described  here  is  0.22  *  3869  =  851  seconds,  ~  14  minutes.  Thus,  on  average,  Hierarchy  groups  require  14 
additional  minutes  to  complete  the  task  over  Edge  counterparts,  with  a  total  task  period  of  37  to  60  minutes. 
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difference.  A  one-way  ANOVA  reveals  that  the  difference  in  the  means  for  time ,  is  significant  (p<0.001 ), 
while  the  difference  in  the  means  for  accuracy  is  not  (p>0.38). 

Given  these  results,  Hypothesis  1  is  partially  supported.  Persons  working  in  an  Edge 
organization,  on  average,  perform  their  work  faster.  However,  persons  working  in  Edge  and  Hierarchy 
configurations  produce  equally  accurate  results.  This  finding  leads  to  our  first  proposition. 

Proposition  1.  For  compiex  tasks  requiring  high  coiiaboration  among  a  mid-sized  group  of 
professional  knowledge  workers ,  persons  working  in  Edge  organizations ,  on  average , 
perform  faster  than  persons  within  Hierarchy  organizations ,  with  no  discernible  change  in 
accuracy. 

In  short,  the  Edge  outperforms  the  Hierarchy,  given  the  same  complex  task  (see  [102,  103]  for  a 
discussion  objective  task  complexity).  However,  we  suggest  further  experimentation  be  undertaken  to 
assess  whether  this  postulate  can  be  generalized  to  other  types  of  tasks  and  task  environments. 

Exchange  of  Mental  Models 

We  then  again  apply  univariate  ANOVA  to  compare  the  sample  performance  means  related  to  the 
exchange  or  lack  of  exchange  of  mental  models.  We  note  that  the  mean  time  for  subjects  in  groups  that 
exchanged  mental  models  is  0.37  (slower),  while  the  mean  time  for  subjects  in  groups  that  did  not 
exchange  mental  models  is  0.45  (faster).  We  also  note  that  mean  accuracy  for  subjects  in  groups  that 
exchanged  mental  models  is  0.72  (more  accurate),  while  the  mean  accuracy  for  subjects  that  did  not 
exchange  mental  models  is  0.61  (less  accurate).  Both  differences  in  the  means  are  significant  (p<0.01). 
For  organizational  designers,  this  result  suggests  that  relative  to  complex  tasks  requiring  high 
collaboration,  a  trade  space  for  performance  exists  between  allowing  organizational  members  to  share 
only  information  (operationalized  as  “factoids”  about  the  impending  terrorist  attack)  and  allowing  them  to 
share  information  and  knowledge  (operationalized  as  person’s  emerging  mental  model  of  the  terrorist 
attack).  While  the  sharing  of  mental  models  results  in  a  more  accurate  performance,  the  sharing  of 
mental  models  also  results  in  a  slower  performance. 

However,  a  more  careful  examination  of  the  data  suggests  that  for  persons  in  a  Hierarchy 
configuration,  the  exchange  of  mental  models  slows  performance  on  average  (mean  time  is  0.47-0.25  = 
0.22  slower  with  exchange  of  mental  models,  p<0.01)  but  also  increases  accuracy  (mean  accuracy  is 
0.69-0.60  =  0.09  more  accurate  with  exchange  of  mental  models,  p<0.08)  Yet  the  Edge  configuration 
appears  sufficiently  robust  such  that  it  can  absorb  the  added  work  of  exchanging  mental  models  and 
thereby  gain  improved  accuracy  (mean  accuracy  is  0.75-0.61  =0.14  more  accurate  with  exchange  of 
mental  models,  p<0.02)  without  suffering  slower  performance  (mean  time  is  0.47-0.45  =  0.02  slower  with 
exchange  of  mental  models,  p>0.55).  Thus  our  preliminary  work  suggests  the  following  propositions. 

Proposition  2.  For  complex  tasks  requiring  high  collaboration  among  a  mid-sized  group  of 
professional  knowledge  workers ,  persons  given  the  opportunity  to  exchange  mental 
models  about  the  task  solution  provide  more  accurate  results. 

Proposition  2a.  For  persons  operating  within  the  Edge ,  the  exchange  of  mental  models 
enhances  accuracy  without  degrading  speed. 

Proposition  2b.  For  persons  operating  within  the  Hierarchy ,  the  exchange  of  mental 
models  degrades  speed  while  improving  accuracy. 

We  caution  the  reader  that  given  the  emphasis  on  accuracy  in  the  instructions  to  the  subjects  (and  by 
extension,  the  incentive  structure  posed  to  the  subjects),  proposition  2b  requires  further  evaluation.4  To 
spare  the  reader,  detailed  statistical  discussion  is  not  provided  as  we  test  our  remaining  three 
hypotheses.  Additional  detail  can  be  obtained  from  the  authors. 


4  The  careful  reader  will  also  note  that  in  the  above  calculations,  we  ignored  subjects  who  failed  to  respond.  If  these 
24  cases  were  included  with  an  assignment  of  0  for  time  (slowest)  and  0  for  accuracy  (least  accurate),  the  levels  of 
significance  are  nearly  identical,  suggesting  that  our  results  are  robust  across  two  very  reasonable 
operationalizations  of  the  data. 
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Professional  Competence 

Our  second  hypothesis  suggests  that  organizations  comprised  of  people  with  greater  professional 
competence  will  outperform  those  with  less  competent  people,  regardless  of  organizational  form. 

At  the  individual  level  of  analysis,  two  proxies  for  operationalizing  professional  competence  are  1 ) 
subjects’  prior  familiarity  with  intelligence  analysis  and  2)  subjects’  years  of  work  experience. 

Intelligence  experience.  Of  our  observations,  48  are  associated  with  persons  who  self-report 
some  experience  with  intelligence  functions,  while  137  are  associated  with  persons  who  self-report  no 
experience  with  intelligence  functions.  We  omit  the  49  cases  for  which  no  data  about  experience  with 
intelligence  functions  is  available.  A  MANOVA  reveals  no  significant  difference  in  the  performance  of 
those  possessing  or  lacking  intelligence  experience. 

Work  experience.  We  next  correlate  work  experience  (in  years)  against  both  performance 
factors,  time  and  accuracy.  There  exists  a  weak  positive  correlation  between  work  experience  and  speed 
(r  =  0.17,  p<  0.02)  and  a  weak  negative  correlation  between  work  experience  and  accuracy  ( r  =  -0.18,  p 
<0.02).  These  findings  suggest  that  more  experienced  workers  identify  the  details  about  the  terrorist 
attack  more  quickly,  but  are  more  prone  to  inaccurate  identifications. 

We  also  consider  Hypothesis  2  at  the  organizational  level  of  analysis.  In  the  context  of  this 
experimentation,  we  can  consider  professional  competence  as  experience  with  command  and  control, 
and  we  compare  the  average  performance  of  Group  B  (undertaking  advanced  C2  coursework)  against 
average  performance  of  Group  C  (undertaking  introductory  C2  coursework)  in  order  to  test  this 
hypothesis.  Since  we  wish  to  compare  the  groups  as  a  whole,  and  not  just  individuals,  we  include  data 
from  subjects  who  did  not  identify  in  the  analysis  as  time  =  0  and  accuracy  =  0.  Again,  a  MANOVA 
reveals  no  significant  difference  in  performance  between  Group  B  and  Group  C.  Thus  Hypothesis  2  has 
mixed  support  depending  on  the  proxy  used  for  professional  competence  and  the  level  of  analysis 
explored. 

We  caution  the  reader,  however,  that  all  of  the  subjects  hold  baccalaureate  degrees,  and  nearly 
all  possess  or  are  working  toward  a  masters  or  doctoral  degree.  Further,  the  selection  process  for 
attending  or  teaching  at  the  university  is  competitive,  indicating  that  all  subjects  are  likely  to  possess  a 
level  of  professional  competence  equal  to  or  surpassing  various  peer  groups.  We  thus  acknowledge 
mixed  support  for  this  hypothesis,  but  suggest  that  further,  more  specific  testing  is  required  to  ensure 
adequate  controls  and  manipulations  for  professional  competence  can  be  incorporated  into  the  ELICIT 
campaign  of  experimentation. 

Task  Repetition 

Hypothesis  3  suggests  that  as  organizations  repeat  tasks,  performance  will  improve,  regardless  of 
organizational  configuration.  We  expect  that  time  and  accuracy  within  our  four  experimental  groups  will 
improve  during  each  round  of  experimentation  -  i.e.,  performance  in  Round  2  will  improve  over  Round  1, 
performance  in  Round  3  will  improve  over  Round  2,  and  so  on. 

Interestingly,  our  plots  for  all  four  Groups  (see  Appendix  C)  challenge  this  hypothesis  -  in  three  of 
our  four  groups  (Groups  A,  B,  and  C),  time  improves  mildly  between  Rounds  1  and  2  and  Rounds  3  and 
4,  but  degrades  mildly  between  Rounds  2  and  3.  Only  one  group  (Group  D)  shows  improvement  or  near 
similarity  for  time  between  Round  2  and  Round  3.  For  accuracy,  the  graph  is  even  less  illuminating. 
Groups  A  and  D  appear  to  hover  at  about  the  same  accuracy  throughout  the  experiment,  with  slight 
degradation  or  improvement  from  one  round  to  the  next.  Group  B  improves,  degrades  and  then 
improves,  while  Group  C  degrades,  improves,  and  improves. 

We  are  interested  in  each  group’s  performance  relative  to  itself  over  time,  so  we  isolate  the  data 
and  perform  four  unique  MANOVA  calculations  followed  by  Scheffe  post-hoc  analyses,  one  for  each 
group  and  including  non-answers  (time  =  0,  accuracy  =  0).  Thus  time  and  accuracy  are  the  dependent 
variables  and  round  of  play  is  the  independent  variable.  For  groups  that  play  all  rounds  in  either  the 
Hierarchy  (Group  D)  or  Edge  (Group  A)  configuration,  the  differences  in  time  are  not  significant.  In  Group 
C,  the  degradation  in  time  when  the  group  switches  from  Edge  to  Hierarchy  (from  0.50  to  0.15,  -22.5 
minutes  slower  over  Round  2)  is  significant  (p<0.05)  when  compared  to  all  other  rounds  of  play.  Similarly, 
in  Group  B,  the  improvement  in  time  when  the  group  switches  from  Hierarchy  to  Edge  (from  0.16  to  0.40, 
-14  minutes  faster)  is  significant  (p<0.05),  as  is  Group  B’s  subsequent  improvement  in  time  between 
Round  3  and  4  (from  0.40  to  0.64,  -  15  minutes  faster). 
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The  results  are  thus  mixed.  If  the  organizational  form  is  unchanging,  the  time  of  the  group 
changes  negligibly.  However,  if  the  organizational  configuration  changes  from  Edge  to  Hierarchy,  the 
group  suffers  performance  degradation  (i.e.,  a  reconstitution  period)  before  recovering  to  its  “normal”  level 
of  performance.  If  the  organizational  configuration  switches  from  Hierarchy  to  Edge,  the  group  improves 
at  the  point  of  the  configuration  change  and  continues  to  improve  subsequently--not  only  is  no  recovery 
period  required  when  transitioning  from  Hierarchy  to  Edge,  but  the  transformation  results  in  improved 
performance  immediately  that  continues  into  subsequent  tasks,  at  least  relative  to  time.  These  findings 
lead  us  to  reject  Hypothesis  3  and  replace  it  with  the  following,  more  refined  postulates. 

Proposition  3a.  Over  time,  neither  Edge  nor  Hierarchy  organizations  improve  their  speed 
on  complex,  highly  cognitive  tasks  requiring  significant  collaboration. 

Proposition  3b.  Edge  organizations  that  are  re-configured  into  Hierarchy  organizations 
suffer  a  significant  degradation  in  speed  before  recovering  to  previous  levels  of 
performance. 

Proposition  3c.  Hierarchy  organizations  that  are  re-configured  into  Edge  organizations 
enjoy  immediate  and  continual  improvements  in  speed,  with  no  need  for  a  recovery 
period. 

On  the  accuracy  dimension  of  performance,  a  slightly  different  pattern  emerges.  In  the  group  that 
plays  the  game  in  the  Edge  configuration  all  four  times  (Group  A),  accuracy  improves  from  Round  1  to 
Round  2  (p<0.05),  degrades  from  Round  2  to  Round  3  (p<0.05)  and  improves  from  Round  3  to  Round  4 
(p  not  significant).  This  group  struggles  with  the  Edge  configuration,  and  reaches  its  peak  performance 
with  respect  to  accuracy  during  Round  2.  In  the  group  playing  in  the  Hierarchy  configuration  all  four  times 
(Group  D),  accuracy  improves  steadily  over  time,  with  significant  improvement  (p<0.05)  from  Round  1  to 
Round  2.  For  Group  C,  the  pattern  of  performance  found  with  time  repeats  itself  for  accuracy  -  i.e., 
accuracy  improves  from  Round  1  to  Round  2  when  the  Edge  configuration  remains  stable,  degrades  at 
the  transformation  to  Hierarchy  (Round  2  to  Round  3),  and  then  recovers  to  its  previous  level  of 
performance  during  Round  4  while  remaining  in  the  Edge  configuration.  All  changes  are  significant.  For 
Group  B,  only  the  improved  accuracy  between  Rounds  1  and  4  is  significant  (p<0.05).  This  finding 
indicates  that  Group  B  shows  similar  or  improved  performance  over  all  four  rounds,  despite  the 
transformation  from  Hierarchy  to  Edge  at  the  midpoint.  Since  we  are  concerned  that  our  strict 
interpretation  of  accuracy  may  affect  these  results,  we  repeat  the  analysis  using  a  relaxed  coding  schema 
for  accuracy.  The  results  for  the  second  analysis  are  generally  similar  (see  Appendix  C  for  details).  We 
thus  feel  confident  offering  the  following  refinements  to  Hypothesis  3. 

Proposition  3d.  Over  time,  both  Edge  and  Hierarchy  organizations  will  improve  their 
accuracy  on  complex,  highly  cognitive  tasks  requiring  significant  collaboration. 

Proposition  3e.  Edge  organizations  that  are  re-configured  into  Hierarchy  organizations 
suffer  a  significant  degradation  in  accuracy  before  recovering  to  previous  levels  of 
performance. 

Proposition  3f.  Hierarchy  organizations  that  are  re-configured  into  Edge  organizations 
absorb  the  change  without  any  degradation  in  accuracy,  and  then  continue  to  improve 
over  time. 

Combining  propositions  3b  and  3e,  as  well  as  3c  and  3f,  leads  to  the  following  propositions. 

Proposition  3g.  Edge  organizations  that  are  transformed  into  Hierarchy  organizations 
suffer  a  significant  degradation  in  performance  before  recovering  to  previous  levels  of 
performance. 
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Proposition  3h.  Hierarchy  organizations  that  are  transformed  into  Edge  organizations 
immediately  operate  faster,  without  any  degradation  in  accuracy,  and  subsequently 
continue  to  improve  on  both  aspect  of  performance. 

We  note,  too,  that  the  two  groups  playing  the  same  organizational  configuration  four  times  (i.e.,  E-E-E-E 
&  H-H-H-H)  generally  improve  from  one  round  to  the  next,  although  the  Edge  group  performs 
exceptionally  and  unexpectedly  well  in  Round  2.  We  defer  comments  on  this  observation  until  the 
discussion. 

Organizational  Learning 

Our  final  hypothesis  suggests  that  performance  by  an  Edge  organization  will  improve  more  quickly  than 
that  of  a  Hierarchy  organization.  We  operationalize  learning  as  the  improvement  (or  degradation)  in 
average  performance  from  one  round  of  play  to  the  next,  given  the  same  organizational  configuration. 

For  example,  if  an  Edge  Group  performed  at  a  time  of  0.24  during  Round  1  and  a  time  of  0.36  during 
Round  2,  the  learning  would  be  measured  as  0.36  -  0.24  =  0.12,  or  an  improvement  of  0.12  during  the 
later  round.  In  the  experimentation,  we  have  five  observations  that  reflect  learning  by  Edge  groups  and 
five  observations  that  reflect  learning  by  Hierarchy  groups. 

As  expected,  the  Edge  groups  learn  slightly  faster  than  the  Hierarchy  groups,  as  evidenced  by 
performance  on  both  the  time  and  accuracy  dimensions  (p:  0.064  >  0.05  for  time,  p\  0.188  >  0.158  for 
accuracy).  However,  these  differences  are  not  significant  (p>0.80),  leading  us  to  reject  Hypothesis  4.  We 
also  note  that  in  terms  of  accuracy,  the  Edge  group  experiences  greater  volatility  than  the  Hierarchy  (a: 
0.28  >  0.12),  leading  us  to  offer  the  following  propositions: 

Proposition  4.  Over  the  same  period  of  time,  Edge  and  Hierarchy  organizations  achieve 
approximately  the  same  level  of  learning. 

Proposition  4a.  Compared  to  Hierarchy  counterparts,  Edge  organizations  may  experience 
greater  volatility  as  they  learn. 

DISCUSSION 

In  this  section,  we  discuss  implications  of  the  results  of  our  laboratory  experiments  for  both  organizational 
design  and  future  research. 

Implications  for  Organizational  Design 

The  Edge  [1,39]  is  an  emerging  concept  within  organizational  design,  and  our  companion  piece  [38]  to 
this  work  suggests  that  the  Edge  resembles  and  performs  as  a  composite  of  other  forms,  notably 
elements  of  Mintzberg’s  [37]  Adhocracy,  Professional  Bureaucracy  and  Simple  Structure.  The  Edge’s  low 
centralization,  low  formalization,  and  high  probability  of  lateral  communication  suggests  it  may  share 
critical  characteristics  with  network  organizations  [48],  particularly  as  viewed  through  the  lens  of 
organizational  information-processing  theory  [82-84],  Indeed,  our  laboratory  experimentation  offers  an 
instantiation  of  organizational  information-processing  theory  in  an  experimental  laboratory  environment 
with  human  subjects,  It  complements  existing  computational  [24,33,34,38]  and  field  research  [70]  on 
Edge  organizations.  Specifically,  we  manipulate  organizational  form  by  imposing  or  removing  controls  on 
the  manner  in  which  subjects  are  permitted  to  access,  share  or  exchange  information  (e.g.,  “factoids”) 
and  knowledge  (e.g.,  “postcards”). 

Comparing  Results  of  Computational  and  Laboratory  Experimentation 

Our  laboratory  experiments  suggest  that  generally  speaking,  Edge  organizations  outperform  Hierarchy 
organizations  when  given  similar  tasks  and  opportunities  for  organizational  learning.  Learning  within 
Edge  organizations  can  be  more  volatile  and  inconsistent  than  in  Hierarchy  organizations,  but  on 
average,  the  two  organizational  forms  learn  at  the  same  pace.  Our  experimentation  thus  intimates  that 
for  complex  tasks  requiring  high  interdependence  and  high  cognitive  skill,  the  Edge  organization  is  the 
superior  form.  More  specifically,  in  our  two-dimensional  performance  model,  the  Edge  outperforms  the 
Hierarchy  on  time  without  sacrificing  accuracy,  while  improving  its  performance  at  an  approximately 
equivalent  rate.  This  finding  buttresses  the  validity  of  our  previous  and  current  computational  modeling, 
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which  suggests  that  the  Edge  outperforms  other  organizational  configurations  but  with  higher  project  and 
function  risk,  resulting  in  more  volatile,  but  superior,  results  than  when  a  Hierarchy  performs  similar  tasks. 

This  work  also  extends  our  previous  computational  experimentation  by  examining  the  relationship 
between  the  exchange  of  information  and  knowledge  and  performance  within  the  two  organizational 
forms.  Our  laboratory  experimentation  intimates  that  the  Hierarchy  organizations  may  suffer  performance 
degradation  when  its  members  are  asked  to  share  both  information  and  knowledge;  specifically,  the 
Hierarchy’s  performance  lessens  when  its  members  are  asked  to  share  their  interpretations  of  available 
information  (\.e.,  knowledge)  rather  than  simply  share  available  information.  For  the  Edge,  however, 
sharing  information  and  knowledge  does  not  result  in  performance  degradation — in  fact,  performance  is 
enhanced.  While  preliminary,  our  results  continue  to  extend  the  margins  of  information-processing  theory 
by  suggesting  that  the  content  of  the  information  to  be  processed  by  an  agent  may  affect  the  observed 
result  of  processing  efforts.  Agents  that  exchange  filtered,  synthesized  and  summarized  information  may 
outperform  those  working  with  information  that  has  not  been  arranged  into  a  more  coherent  whole. 

Evaluating  Performance 

To  compare  the  performance  of  our  four  organizational  form-mental  model  pairs,  organizational 
designers  may  find  it  helpful  to  envision  the  two  dimensions  of  performance  on  a  grid,  with  time  on  the  x- 
axis  and  accuracy  on  the  y-axis  (Figure  3).  Three  unique  y  =  —x  +  b  indifference  curves  are  included, 

with  b1}  b2,  and  b3  as  unspecified  constants.  The  indifference  curves  represent  levels  of  relative 
performance  (i.e.,  closest  to  the  origin  =  worst  performance,  farthest  from  the  origin  =  best  performance) 
in  which  the  evaluator  has  equally  weighted  time  and  accuracy  (e.g.,  0.10  improvement  in  time  is  equal  to 
0.10  improvement  in  accuracy).  Any  point  along  an  indifference  curve  would  be  considered  as  providing 
comparable  performance.  In  such  an  evaluation  space,  the  Hierarchy  with  the  exchange  of  mental 
models  underperforms  all  others,  while  the  Edge  with  the  exchange  of  mental  models  outperforms  all 
others.  When  mental  models  are  not  exchanged,  our  observations  suggest  that  the  Edge  and  Hierarchy 
would  fall  along  a  nearly  similar  indifference  curve  -  i.e.,  they  are  interchangeable  in  terms  of  observed 
performance.  While  preliminary,  this  finding  suggests  that  for  complex  tasks  in  which  the  network 
architecture  allows  only  for  the  formal  or  informal  exchange  of  unprocessed  information,  neither  the  Edge 
nor  Hierarchy  outperforms.  For  complex  tasks  in  which  the  network  architecture  allows  for  sharing  of 
interpretations  of  this  information,  the  Edge  outperforms  the  Hierarchy. 

We  caution  that  the  indifference  curves  described  here  are  useful  for  discussing  and  evaluating 
organizational  performance,  but  are  tautological  in  nature.  The  curves  provide  a  means  for  evaluating  the 
observed  performance  of  various  organizational  form-mental  model  pairings  relative  to  each  other  and 
laden  with  stakeholder  values  time  and  accuracy  as  equally  important  performance  factors.  If  values 
associated  with  the  performance  factors  were  changed  by  the  stakeholders,  one  might  draw  different 
comparative  conclusions.  Also,  our  data  offers  no  reason  to  suspect  that  the  relationship  between  time 
and  accuracy  is  linear,  and  in  fact,  contradicts  any  such  assertion. 


Figure  3:  Evaluating  Observed  Performance  via  Indifference  Curves 
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Measuring  the  Cost  of  Organizational  Transformation 

One  of  the  more  important  aspects  of  our  experimental  protocol  is  providing  a  means  to  measure  the 
switching  cost  associated  with  organizational  transformation — i.e.,  initiating  exploration  of  the  extent  by 
which  performance  is  degraded  when  an  organization  switches  from  Edge  to  Hierarchy,  or  from  Hierarchy 
to  Edge.  The  research  thus  demonstrates  a  complexity  beyond  simple  comparison  of  two  organizational 
forms  relative  to  organizational  performance — it  provides  an  empirical  investigation  of  the  cost  of 
dynamically  reconfiguring  an  organization  at  the  midpoint  of  its  life  cycle.  Our  results  are  both  surprising 
and  illuminating.  While  one  would  expect  that  dynamically  reconfiguring  an  organization  would  generally 
lead  to  a  degraded  performance  followed  by  recovery  to  previous  performance  levels  after  some  time  lag, 
we  discover  that  this  pattern  occurs  only  when  the  Edge  reconfigures  into  Hierarchy,  not  when  the 
Hierarchy  reconfigures  into  an  Edge.  The  Hierarchy,  reconfigured  into  an  Edge,  suffers  no  degradation  in 
performance  (as  defined  by  time  and  accuracy),  and  in  fact,  appears  to  thrive  and  excel  in  its  new  Edge 
configuration. 

The  implications  of  these  two  results  could  conceivably  have  extensive  ramifications  for  the 
design  of  command  and  control  processes,  and  deserve  both  further  experimental  scrutiny  and  careful 
consideration  by  policymakers.  Consider,  as  an  example,  disaster  relief  operations.  For  those  at  or  first 
to  arrive  on  the  scene  of  a  disaster,  rescue  and  relief  efforts  are  often  characterized  as  chaotic,  uncertain, 
and  complex.  Previous  and  concurrent  computational  modeling  suggests  Edge  configurations  are 
preferable  in  such  contexts  [34,38].  During  relief  operations,  stakeholders  surface  to  make  demands  on 
already  stretched  resources,  while  service  providers  arrive  and  attempt  integrate  their  capabilities  into 
ongoing  rescue/relief  efforts  [85-87].  Although  military  organizations  have  long  been  involved  in  disaster 
relief  efforts  [88-90],  few,  if  any,  laboratory  experiments  have  informed  senior  defense  officials  as  more 
optimal  command  and  control  structures  for  integrating  military  capabilities  into  disaster  relief  operations. 
While  nascent  and  preliminary,  our  empirical  investigations  suggest  that  given  overarching  goal  clarity, 
military  units  assigned  to  disaster  relief  missions  and  vested  with  Edge-like  characteristics  could 
outperform  similar  military  units  assigned  to  disaster  relief  missions  and  managed  via  a  more  traditional 
Hierarchy  command  and  control  structure.  Certainly,  this  assertion  is  broad  and  sweeping,  and 
instantiation  of  Edge-like  command  and  control  processes  versus  Hierarchal  command  and  control 
processes  for  military  units  requires  further  reflection  and  discourse  that  would  not  prove  insignificant.  At 
an  abstract,  basic  level,  however,  our  findings  suggest  that  wholly  antithetical  thinking  about  traditional 
military  command  and  control  processes  may  be  a  useful  step  toward  designing  high-performance 
organizations  for  complex,  chaotic  environments. 

Implications  for  Future  Research 

The  ELICIT  laboratory  environment,  coupled  with  robust  research  design,  offers  an  opportunity  for 
researchers  to  explore  organizational  dynamics  in  microscopic  detail,  particularly  at  the  size  of  a  mid¬ 
sized  workgroup  or  even  small  firm.  With  a  sizable  portion  of  experimental  findings  about  team 
performance  derived  from  experimentation  on  groups  sized  at  two  to  nine  individuals  ([104-109]), , 
incorporating  17  players  in  the  ELICIT  environment  provides  laboratory  experimentation  with  group  sizes 
that  have  generally  been  reserved  for  field  studies  (see,  e.g.,  cf.  [110]  for  a  field  study  involving  multi¬ 
sized  groups  from  under  5  to  over  20)  or  computational  experimentation  (see,  e.g.,  [38]).  Log  files 
automatically  scribe  essential  data  that  could  assist  researchers  with  exploring  hypotheses  from  a 
plentitude  of  theoretical  domains,  and  additional  data  collection  can  complement  software  logs  as  the 
research  design  requires.  In  this  section,  we  offer  some  initial  thoughts  about  extending  this  research 
program. 

Edge  vs.  Hierarchy 

The  Hierarchy,  often  considered  an  instantiation  of  Weber’s  [69]  bureaucratic  structures  and  Mintzberg’s 
[37]  Machine  Bureaucracy,  is  a  long-studied  phenomenon  within  social  science.  Yet  other  forms  of 
organizing  clearly  exist,  and  the  study  of  these  organizational  forms  materializes  as  a  diverse  and 
sometimes  even  contentious  stream  of  research  within  organizational  studies  [e.g.,  47,48,49,50],  Further, 
recent  advances  in  our  conceptions  and  instantiations  of  self-organization  [91 ,92],  complexity  theory, 
worker/employee  empowerment  [93]  and  innovation  labs  [94]  point  to  a  growing  body  of  empirical  studies 
and  theoretical  frameworks  that  can  serve  to  inform  investigation  of  Edge,  or  Edge-like,  organizations. 

The  research  presented  in  this  work,  coupled  with  our  complementary  computational  experimentations, 
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draws  from  and  extends  upon  these  traditions,  and  we  ground  our  efforts  in  organizational  theory  by 
instantiating  differences  between  the  two  organizational  forms  from  an  information-processing 
perspective  [52].  We  thus  contend  that  one  manner  in  which  researchers  can  differentiate  organizational 
forms  is  through  a  detailed  understanding  of  how  the  agents  within  each  form  receive,  process,  store  and 
exchange  available  information. 

The  ELICIT  framework  is  particularly  suited  to  investigations  grounded  in  this  framework,  as  the 
primary  manipulations  relate  to  the  exchange  and  storage  of  information,  and  to  some  extent,  knowledge, 
among  the  participants.  The  results  presented  in  this  paper,  however,  merely  scratch  the  metaphorical 
surface  of  the  investigations  made  possible  with  the  data,  and  in  particular,  offer  coarse  (albeit  important 
baseline)  insights  into  how  the  Edge  performs  compared  to  a  more  traditional  and  well-known 
organizational  form,  the  Hierarchy.  Realizing  the  full  potential  of  this  information  processing  perspective 
toward  organizational  design  in  general  and  comparing  organizational  forms  in  particular,  however, 
requires  a  commitment  to  cogently  expressing  high-level,  but  often  nebulous,  concepts  into  experimental 
observables.  For  example,  the  terms  “decision  rights”  and  “distribution  of  power”  are  often  articulated  as 
varying  among  organizational  forms.  However,  how  these  and  like  concepts  can  (and  should)  be 
instantiated  in  the  ELICIT  environment  remains  an  area  for  future  consideration.  We  are  hopeful, 
however,  that  our  results  serve  as  encouragement  for  considering  the  ELICIT  laboratory  experimentation 
as  an  opportunity  for  both  mid-range  theory  testing  and  finely  honed  micro-theorizing,  particularly  as 
these  investigations  relate  to  Hierarchy  and  Edge  forms. 

Incentives 

The  relationship  between  incentive  structures  and  both  organizational  performance  and  organizational 
learning,  for  example,  represent  major  unknowns  with  regard  to  Edge  organizations.  Our  analysis 
considers  time  and  accuracy  as  separate  dimensions  of  performance,  and  we  normalize  both  factors 
across  all  16  groups.  We  discover  that  variance  in  time  from  one  round  to  the  next  for  the  two  groups  that 
played  a  single  organizational  configuration  consistently  (i.e.,  Edge-only  or  Hierarchy-only)  is  not 
significant.  At  first  glance,  these  findings  appear  surprising,  but  we  attribute  the  results  as  evidence  that 
our  incentive  structure  emphasizes  accuracy  over  time.  A  subject  “wins”  the  game  if  he  or  she  offers  the 
most  correct  solution  among  his  or  her  peers,  regardless  of  how  quickly  he  or  she  provides  the  details 
about  the  terrorist  attack.  The  findings  suggest  that  unearthing  incentives  that  emphasize  accuracy  and 
time  in  mid-sized  groups  is  a  challenging  task,  and  our  work  to  date  requires  significant  refinement. 
However,  the  results  reported  in  this  paper  offer  a  baseline  against  which  other  experimentation  on 
incentive  structures  could  be  referenced. 

Motivation  and  Organizational  Learning 

For  our  experimentation,  evidence  of  organizational  learning  thus  becomes  primarily  instantiated  in 
improved  accuracy-  i.e.,  since  is  time  is  relatively  stable,  we  expect  each  group's  accuracy  to  improve 
over  the  previous  round  of  play  as  evidence  of  group  learning.  In  both  our  strict  and  relaxed  criteria,5  the 
Edge-only  and  Hierarchy-only  groups  demonstrate  a  steady  improvement  in  accuracy,  excepting  Round  2 
with  the  Edge  group.  In  this  one  case,  the  accuracy  appears  abnormally  high  compared  to  measured, 
steady  improvement  (for  detail,  see  Figure  4  in  Appendix  C).  One  plausible  explanation  for  this  spike  in 
improved  performance  is  the  possibility  that  ELICIT  version  3  is  less  difficult  compared  to  the  others.  This 
assertion  is  buttressed  by  observing  that  two  of  our  four  groups  demonstrate  peak  performance  when 
playing  version  3,  and  yet  is  contradicted  by  two  of  the  four  groups  demonstrating  their  peak  performance 
during  play  of  version  1.  Another  plausible  explanation  is  that  if  performance  is  related  to  motivation, 
perhaps  motivation  is  highest  once  the  purpose  and  format  of  the  game  is  well  understood  (i.e.,  when 


5  Generally  speaking,  the  relaxed  criteria  described  here  accepts  linguistically  similar  answers  as  correct  across  all 
four  rounds,  and  thus  ignores  some  of  the  nuance  available  in  current  versions  of  ELICIT.  For  example,  in  two  of  the 
ELICIT  versions,  articulating  the  fully  correct  answer  of  the  intended  target  of  the  terrorist  attack  (i.e.,  the  “what”) 
involves  subjects  providing  a  modifier-noun  combination  (e.g.,  “corporate  headquarters,”  “subsidiary  headquarters”) 
instead  of  providing  an  unmodified  noun  (e.g.,  “headquarters”).  Under  our  strict  schema,  the  answer  of 
“headquarters,”  if  the  correct  answer  is  “subsidiary  headquarters,”  is  considered  incorrect.  Under  our  relaxed 
schema,  “headquarters”  is  considered  correct.  The  relaxed  schema  thus  provides  greater  between-group 
consistency  for  evaluating  subject  responses  among  all  four  ELICIT  versions.  The  experimental  trade  off  is  reducing 
our  ability  to  explore  how  the  various  groups  respond  to  variance  in  task  complexity. 
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task  clairty  is  high)  and  the  experience  is  still  somewhat  novel  (i.e.,  when  the  task  appears  non-routine 
and/or  out  of  ordinary) — i.e.,  during  Round  2,  when  version  3  of  the  game  is  played.  We  leave  the 
answers  to  such  questions  to  future  researchers,  but  offer  that  the  results  presented  in  this  work  could 
serve  as  a  baseline  for  a  family  of  research  questions  related  to  the  relationship  between  motivation 
theory  and  performance. 

CONCLUSION 

The  Edge  is  a  relative  newcomer  to  organizational  design,  but  it  is  emerging  as  an  important  organizing 
concept  for  contemporary  military  operations.  Little  is  known,  however,  about  the  performance  of  Edge 
organizations  relative  to  alternatives,  leading  to  the  current  campaign  of  Edge  research.  The  campaign 
investigates  the  Edge  using  complementary  computational,  field,  and  laboratory  techniques,  and  the  work 
presented  here  offers  preliminary  analysis  of  our  laboratory  experimentation.  The  experiments  lead  us  to 
refine  our  hypotheses  about  the  Edge,  its  learning  curve,  and  its  performance  through  direct  comparison 
against  the  better-known  Hierarchy  in  a  laboratory  setting.  The  laboratory  setting  allows  us  to  adequately 
control  for  many  variables — such  as  rank,  gender,  assignment  of  persons  to  particular  roles,  allowable 
patterns  of  communication — that  field  experimentation  does  not.  Our  results  are  thus  based  on  the 
carefully  executed  manipulations  of  experimental  research. 

In  the  course  of  experimentation,  we  discover  unexpected  attributes  of  both  Edge  and  Hierarchy 
organizations,  including  empirical  measures  of  the  costs  associated  with  transforming  from  one  form  to 
another.  In  so  doing,  we  contribute  an  empirical  baseline  for  ongoing  discussions  about  resource 
ramifications  associated  with  dynamically  transforming  organizations.  Parts  of  our  analysis  confirm  the 
results  of  earlier  and  companion  computational  experimentation  -  chiefly,  that  Edge  organizations 
outperform  Hierarchy  organizations  in  certain  task  and  environmental  contexts,  and  in  terms  of  certain 
performance  measures.  Yet,  our  experimentation  also  serves  to  inform  future  computational 
experimentation  and  extend  existing  computational  models  through  our  observations  that:  1)  Edge  and 
Hierarchy  organizations  learn  at  the  same  rate,  but  the  Edge  experiences  greater  volatility  in  its  learning, 
2)  transforming  from  Hierarchy  to  Edge  results  in  either  similar  or  improved  performance  on  subsequent 
tasks,  and  3)  transforming  from  Edge  to  Hierarchy  results  in  degraded  performance  initially,  with 
subsequent  recovery  to  previous  levels. 

Beyond  testing  a  series  of  hypotheses  and  refining  our  predictions  about  the  Edge  with  greater 
clarity  and  detail,  we  suggest  that  the  ELICIT  experimental  environment,  coupled  with  a  fastidious 
research  design,  provides  the  opportunity  to  investigate  research  questions  related  to  team 
decisionmaking  at  well  beyond  the  usual  team  investigative  size  of  two  to  six  individuals.  Our  work  is 
grounded  in  information  processing  theory,  and  we  suggest  that  the  ELICIT  environment  is  particularly 
suited  to  this  theoretical  framework.  Our  operationalization  of  important  concepts  (e.g.,  Edge  and 
Hierarchy  in  a  complex  collaborative  task,  bi-dimensional  performance)  contributes  to  both  the  empirical 
and  theoretical  literatures.  It  also  suggests  further  opportunities  for  organizational  theorists  to  define 
more  abstract  concepts  (e.g.,  “decision  rights”  or  “empowerment”)  in  the  context  of  Galbraith’s 
conceptions  about  how  organizations  process  information.  Such  clarification  and  understanding  will 
greatly  assist  efforts  to  bridge  latent  variables  with  experimental  observables,  and  advance  our 
understanding  of  the  internal  organizational  dynamics  of  Edge  and  other  organizational  forms.  We 
recognize  that  other  areas  of  organizational  theorizing  (e.g.,  studies  on  leadership,  motivation,  incentives, 
etc)  could  be  easily  incorporated  into  the  Edge  campaign  of  experimentation,  and  we  hope  we  have 
provided  a  sound  empirical  baseline  against  which  such  discussions  can  continue  to  unfold. 
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APPENDIX  A  -  PARTICIPANT  INSTRUCTIONS  (GROUP  A) 

Instructions 

You  have  been  assigned  to  an  Edge  organization.  Your  goal  is  to  identify  details  about  an  impending 
terrorist  attack.  You  may  communicate  to  other  players  in  two  ways:  1 )  sharing  and  posting  factoids  via 
the  software  posting  factoids  to  websites,  and  2)  sending  “postcards.”  You  may  also  pull  factoids  from 
websites.  Verbal  communication  is  not  permitted  during  the  game. 

Sharing,  posting  and  pulling  factoids  via  the  software 

The  software  supports  two  ways  of  informing  group  members  about  factoids  you  have  “discovered.”  You 
can  Share  a  factoid  directly  with  another  group  member  using  the  Share  tab.  You  can  also  Post  a  factoid 
to  or  Pull  a  factoid  from  any  website.  Other  group  members  can  do  the  same. 

There  are  four  websites:  Who,  What,  Where  and  When.  Though  these  areas  are  called  websites,  the 
information  display  is  provided  by  the  experiment  software  and  not  by  the  Internet. 

•  Factoids  in  your  inbox  can  be  copied  into  your  MyFactoids  list  by  selecting  the  factoid  and  clicking 

on  the  Add  to  MyFactoids  action. 

•  To  Share  a  factoid,  select  the  factoid  from  either  your  inbox  or  your  MyFactoids  list  that  you  wish 
to  share.  Click  on  the  Share  action,  and  select  the  pseudonym  of  the  person  with  whom  you 
want  to  share.  This  sends  the  factoid  to  the  selected  player’s  inbox  message  list. 

•  To  Post  a  factoid,  select  the  factoid  from  your  inbox  or  MyFactoids  list.  Click  on  the  Post  action, 
and  select  the  website  you  wish  to  post  to. 

•  To  Pull  a  factoid,  select  the  factoid  you  wish  to  copy  from  the  website  and  click  on  the  Add  to 
MyFactoids  action.  The  Add  to  My  Factoids  action  can  be  used  to  copy  a  factoid  from  a  website 
to  your  MyFactoids  list. 

Sending  “Postcards” 

Periodically  during  the  experiment,  you  will  be  asked  to  send  a  “postcard”  to  one  other  player  of  your 
choosing.  You  do  NOT  have  to  send  the  postcards  to  the  same  player  each  time.  The  postcard  should 
reflect  your  assessment  of  the  attack  details  at  that  point  in  time.  Your  postcards  must  have  the  following 
format: 


Postcard 


From:  <your  pseudonym> 

Date: 

To:  <addressee’s  pseudonym> 

Time: 

My  assessment  of  the  attack  is: 

Certainty 

Group: 

High 

Moderate 

Low 

None 

Target: 

High 

Moderate 

Low 

None 

Country: 

High 

Moderate 

Low 

None 

Month: 

High 

Moderate 

Low 

None 

Day: 

High 

Moderate 

Low 

None 

Time  of  day: 

High 

Moderate 

Low 

None 

Method  of  attack: 

High 

Moderate 

Low 

None 

22 


Other  software  tools 


Some  other  tools  are  available  to  you  in  the  software: 

•  To  get  a  summary  list  of  all  the  factoids  in  your  MyFactoids  list,  click  on  the  MyFactoids  tab  in  the 
middle  of  your  screen. 

•  To  find  out  your  role  information  and  how  other  members  of  your  group  see  you,  click  on  the 
“How  I’m  seen”  tab. 

•  To  get  a  list  of  all  the  members  in  your  group,  with  information  about  their  role  and  country,  click 
on  the  “What  I  see”  tab. 

•  To  access  information  from  a  team  website,  click  on  the  website  that  you  wish  to  view.  To  update 
the  website  with  the  latest  information  that  has  been  posted  to  it,  click  on  the  Refresh  action  at 
the  top  of  the  screen,  while  viewing  the  website. 

Identifying  the  Who,  What,  Where,  and  When  of  the  Attack 

When  you  think  that  you  have  identified  the  who,  what,  where  and  when  of  the  adversary  attack,  click  on 
the  Identify  tab  at  the  top  of  your  screen  and  enter  free  text  messages  that  identify  the  who,  what,  where 
and  when  of  an  adversary  attack.  Partial  answers  are  accepted,  but  you  may  Identify  only  one  time. 

•  The  who  is  a  group  (for  example  the  blue  group). 

•  The  what  is  a  type  of  target  (for  example  an  embassy  or  religious  school  or  dignitary) 

•  The  where  is  the  country  in  which  the  attack  will  take  place  (for  example  Alphaland) 

•  The  when  is  the  month,  day  and  time  of  day  on  which  the  attack  will  occur  (for  example 
December  15,  at  3:00  a.m.) 

During  the  game 

During  each  experiment  round,  you  are  free  to  work  on  any  aspect  of  the  task. 

Winning  the  Game 

Once  all  the  players  have  identified  a  solution  and  the  surveys  are  complete,  you  will  be  asked  to 
determine  who  emerged  as  the  ‘team  leader’  during  this  round.  Your  selection  should  reflect  a  group 
consensus.  After  the  game  has  been  played,  you  may  talk  amongst  each  other  to  select  the  emergent 
team  leader.  Verbal  communication  is  not  permitted  during  the  game,  but  is  permitted  once  all  of  the 
surveys  are  complete  and  you  are  selecting  your  emergent  team  leader. 

The  games  are  structured  as  a  tournament  that  recognizes  the  contributions  of  both  individuals  and 
groups.  You  receive  one  individual  point  if  you  identify  the  correct  solution  and  your  emergent  group 
leader  identifies  the  correct  solution.  Your  group  receives  a  group  point  if  your  emergent  group 
leader  identifies  the  correct  solution.  After  the  four  games  are  played,  the  points  will  be  totaled.  You 
can  receive  a  maximum  of  four  individual  points  during  the  tournament,  and  your  group  can  receive  a 
maximum  of  four  group  points.  In  the  event  of  an  individual  tie  (e.g.,  1 1  players  identify  four  correct 
solutions  and  the  emergent  group  leader  identifies  the  correct  solution  in  all  cases),  the  fastest  individual 
average  time  to  identify  wins.  In  the  event  of  a  group  tie  (e.g.,  all  group  leaders  identify  the  correct 
solution),  the  fastest  average  time  for  the  group  to  identify  will  win.  Therefore,  it  is  in  your  best  interest 
to  identify  the  correct  solution  as  quickly  as  possible  while  also  ensuring  that  your  emergent 
group  leader  identifies  the  correct  solution  as  quickly  as  possible.  You  may  only  use  the  ‘identify’ 
function  in  the  software  one  time. 
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Game  Over 


The  game  is  over  when  all  players  have  made  their  identification,  or  60  minutes  have  elapsed  (whichever 
is  sooner).  You  will  also  be  asked  to  complete  a  different  short  survey  at  the  end  of  the  experiment. 

Summary 

You  have  been  assigned  to  an  edge  organization.  This  assignment  affects  how  you  can  communicate 
with  other  players. 

•  Sharing  factoids:  You  may  share  factoids  with  any  player  of  your  choosing,  and  you  may  share 
any  single  factoid  as  many  times  as  you  wish.  Factoids  are  shared  via  the  ELICIT  software. 

•  Sending  postcards:  You  may  send  a  postcard  to  any  player  of  your  choosing  at  specified 
intervals. 

•  Posting  to  websites:  You  can  post  any  factoid  to  any  website  of  your  choosing. 

•  Pulling  from  websites:  You  may  pull  factoids  posted  on  any  website. 

You  can  refer  to  these  instructions  during  the  course  of  the  experiment  by  clicking  again  on  the  URL  in 
the  moderator  message. 

When  you  have  finished  reading  this  important  group  background  information  and  are  ready  to  begin, 
click  the  Ready  button  in  the  upper  left  corner  of  your  screen. 

Thank  you  for  playing,  and  good  luck! 


Edge 

1  want  to: 

Share  a  Factoid 

Factoids  can  be  sent  to  any  other  player 
via  the  software 

Send  a  Postcard 

Handwritten  postcards  can  be  sent  to  any 
other  player 

Post  a  factoid  to  a  website 

Factoids  can  be  posted  to  any  website 
via  the  software 

Pull  a  factoid  from  a  website 

Factoids  can  be  pulled  from  any  website 
via  the  software 

Table  3:  Quick  Reference  Table  --  Edge 


Note  1 :  Instructions  were  adapted  from  those  originally  created  by  Parity  Communications 

Note  2:  Groups  B,  C  and  D,  as  listed  in  Table  1 ,  were  provided  short  descriptions  of  Edge  or  Hierarchy 

organizations  in  the  instructions. 
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Appendix  B  -  Subject  Demographics 

ELICIT  Knowledge  Inventory  Questionnaire 


Directions 

Please  complete  this  questionnaire  candidly.  Your  information  will  kept  in  confidence,  and  will  be  used  for 
anonymous  statistical  analysis  and  reporting  purposes  only.  If  you  have  any  questions  or  concerns, 
please  consult  the  Experiment  Moderator. 

Your  name: _ 

Today’s  date: _ 

1 .  What  is  your  age  in  years? _ 

2.  How  many  years  of  work  experience  do  you  have  since  college  graduation? _ 

3.  Please  list  each  college  degree,  major,  institution  and  date  separately  (e.g., 

BS,  Economics,  UC  Berkeley,  2000 

MS  Information  Sciences,  Naval  Postgraduate  School,  2004) 


4.  How  many  years  of  military/government  experience  do  you  have? _ 

5.  In  which  military/government  branch  do  you  work  (e.g.,  Army,  Navy)?  _ 

6.  What  is  your  current  rank  (e.g.,  04/LCDR,  GS13)? _ 

7.  What  is  your  area  of  greatest  military  expertise  (e.g.,  artillery,  aviation) 


8.  How  many  years’  experience  do  you  have  in  this  area? _ 

9.  How  many  years’  experience  in  the  intelligence  field  do  you  have? 

10.  What  is  the  greatest  number  of  people  that  you  have  supervised? 


25 


Appendix  C  -  Performance  by  Experimental  Group 


Figure  4:  Observed  Accuracy  (Strict  Criteria  -  left,  Relaxed  Criteria  -  right),  Groups  A-D 


Round 


-SE-E-E(A)  - 


-  H-H-H-H  (D)  - 


-  E-E-H-H  (C) 


H-H-E-E(B) 


Figure  5:  Observed  Time,  Groups  A-D 
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Appendix  D  -  Performance  by  Primary  Manipulations 


Organizational  Form 

Edge 

Hierarchy 

All 

Time 

p  :  0.47 
o\  0.19 

p  :  0.25 
cr.  0.19 

p  :  0.37 
g:  0.22 

Exchanged 

Accuracy 

p  :  0.75 
o:  0.27 

p  :  0.69 
o:  0.26 

p  :  0.72 
g:  0.26 

No  response 

1 

6 

Mental 

Time 

p  :  0.45 
o:  0.17 

p  :  0.47 
g:  0.10 

p  :  0.45 
g:  0.14 

Models 

Not  Exchanged 

Accuracy 

p :  0.61 

a:  0.35 

p  :  0.60 

g:  0.25 

p  :  0.61 

g:  0.31 

No  response 

11 

6 

All 

Time 

p  :  0.47 
a:  0.18 

p  :  0.33 
g:  0.20 

Accuracy 

p  :  0.69 
a:  0.31 

p  :  0.66 

g:  0.26 

Table  4:  Descriptive  Statistics  for  Two  Dimensional  Performance 


Form 

Mean 

Std.  Deviation 

N 

Edge 

.0640 

.13202 

5 

Learning  -  Time  Hierarchy 

.0500 

.13058 

5 

Total 

.0570 

.12401 

10 

Edge 

.1880 

.27779 

5 

Learning  -  Accuracy  Hierarchy 

.1580 

.12357 

5 

Total 

.1730 

.20331 

10 

Table  5:  Learning  vs.  Organizational  Form 


Experimental 

ELICIT 

Organizational  Form 

Mental  Model 

Group 

Version 

Organizational 

Correlation  Coefficient 

1.000 

.044 

.185** 

-.008 

Form 

Sig.  (2-tailed) 

.523 

.007 

.906 

Correlation  Coefficient 

1.000 

.877** 

.016 

Mental  Model 

Sig.  (2-tailed) 

.000 

.815 

Experimental 

Correlation  Coefficient 

1.000 

.001 

Group 

Sig.  (2-tailed) 

.994 

Correlation  Coefficient 

1.000 

ELICIT  Version 

Sig.  (2-tailed) 

N 

210 

210 

210 

210 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


Table  6:  Spearman's  Rho  Correlation  Coefficients  for  Independent  Variables 
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Appendix  E  -  Operationalization  of  Accuracy 


Correct 

answer 

Subject  1 
identification 

Accuracy  (points) 

Subject  2 
identification 

Accuracy  (points) 

Who 

(Group) 

Tan 

Tan  or  Blue 

0 

Tan 

1 

What 

(Target) 

Omegaland’s 

chancery 

Chancery 

0 

Chancery  of 
Omegaland 

1 

Where 

(Country) 

Betaland 

Omegaland 

0 

Betaland 

1 

When 

(Month) 

June 

June 

1 

May  or  June 

0 

(Date) 

15 

15 

1 

0 

(Time) 

3  p.m. 

1500 

1 

3 

0 

Normalized 

Accuracy 

measure 

o+o+o+(1+1+1> 

A=  3  0.25 

4 

.  ,  ,  (0+0+0) 
1+1+1+- - - 

A=  3  =0.75 

4 

Table  7:  Illustration  of  Accuracy  Measurement  (Strict  Criteria) 


Correct 

answer 

Subject  1 
identification 

Accuracy  (points) 

Subject  2 
identification 

Accuracy  (points) 

Who 

(Group) 

Tan 

Tan  or  Blue 

0 

Tan 

1 

What 

(Target) 

Omegaland’s 

chancery 

Chancery 

1 

Chancery  of 
Omegaland 

1 

Where 

(Country) 

Betaland 

Omegaland 

0 

Betaland 

1 

When 

(Month) 

June 

June 

1 

May  or  June 

0 

(Date) 

15 

15 

1 

0 

(Time) 

3  p.m. 

1500 

1 

3 

0 

Normalized 

Accuracy 

measure 

A  .  .  (i+i+i) 

0  +  1  +  0  +  - - - 

A=  3  =0.50 

4 

..  ,  ,  (0+0+0) 
1+1+1+- - - 

A=  3  =0.75 

4 

Table  8:  Illustration  of  Accuracy  Measurement  (Relaxed  Criteria) 
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Overview 


□  Theoretical  Framework 

■  Contingency  Theory  (Lawrence  &  Lorsch,  1967; 
Burns  &  Stalker,  1961;  Donaldson,  1985) 

■  Information  Processing  Theory  (Galbraith,  1973; 
Tushman  &  Nadler,  1978) 

□  Hypotheses 

□  Experimental  Environment 

□  Initial  Findings 

□  Implications  for  Theory 

□  Implications  for  Practice 

□  Recommendations  for  Future  Experimentation 


Contingency  Theory 


□  Classic  Perspective  (Pugh,  1968;  Hage  &  Aiken,  1969; 
Perrow,  1967) 


Contingency 


Structure 


Performance 


□  Contemporary  Perspective  (Gresov  and  Drazin,  1997; 
Meyer  et  al,  1983) 


Information  Processing  Lens 


□  Organizational  Form  (Structure") 

Edae 

Centralization  Low 

Specialization  Low 

Formalization  Low 

Hierarchy 

High 

High 

High 

Contingency 

□  Knowledge  Sharing,  Exchange  of  Mental  Models 

See:  Galbraith,  1973;  Tushman  &  Nadler,  1978;  Birkinshaw  et 
al,  2002;  Cook  et  al  2000;  Nissen  &  Orr,  2005;  Gateau  et  al, 
2007 


Organizational  Forms 


□  Edge 
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□  Hierarchy 


Who  Group  What  Group  Where  Group  When  Group 


Hypotheses 


□  Hypothesis  1.  People  working  together  in  an  Edge 
organization  will  outperform  those  who  perform  the 
same  work  in  a  Hierarchy. 
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□  Hypothesis  3.  Performance  of  an  Edge  or  Hierarchy 
organization  will  increase  over  time  and  through  task 
repetition. 

□  Hypothesis  4.  Performance  of  an  Edge  organization 
will  increase  more  quickly  than  that  of  a  Hierarchy. 


Experimental  Design 


□  2x2x2  Factorial  Design 

■  Organizational  Form  (Edge,  Hierarchy) 

■  Organizational  Change  (None,  "Switched"  at  midpoint) 

■  Exchange  of  Mental  Models  (Allowed,  Disallowed) 

□  Four  teams  over  four  weeks  undertake  counterterrorism  task 

■  Task  involves  high  cognition  and  interdependence  among  players 

■  Each  team  member  identifies  future  terrorist  attack  only  once 

■  New  scenario  each  week;  all  teams  solve  homomorphic  scenarios 

□  Operationalization  of  Edge  vs.  Hierarchy  information  sharing 

Edge  Hierarchy, 

To  any  player  To  any  player 

To  any  website  To  team  website  only* 

From  any  website  From  team  website  only* 

To  any  player  To  teammate  or 

supervisor  only* 

*  Within  the  Hierarchy,  the  Cross-Team  Coordinator  could  post  &  pull  from  any  website,  as 
well  as  send  "Postcards"  to  subordinates 

□  Objective  performance  measured  by  aggregate  time  and  accuracy 


Share 

Post 

Pull 

"Postcard" 


Experimental  Groups 
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Key: 

V1-V4:  Elicit  Version  1-4 

H:  Hierarchy  manipulation 

E:  Edge  manipulation 

MM:  Mental  models  distributed 

No  MM:  Mental  models  not 
distributed 

Initial  Findings 


□  Organizational  Form 

■  Edge  outperforms  Hierarchy  in  time 

□  Same  accuracy,  greater  speed 

□  Exchange  of  mental  models 

■  Exchanging  mental  models  improves  accuracy 

□  In  Edge,  exchanging  mental  models  improves  accuracy  while 
problem-solving  speed  is  constant 

□  In  Hierarchy,  exchanging  mental  models  improves  accuracy 
while  problem-solving  slows 

□  Organizational  Change 

■  Hierarchy  to  Edge:  No  degradation 

■  Edge  to  Hierarchy:  Significant  degradation,  then  recovery 

□  Organizational  Learning 

■  Edge  and  Hierarchy:  Rate  of  team  learning  is  about  equal 

■  Individual  learning  within  the  Edge  is  more  volatile  than  within 
the  Hierarchy 


Mean  Accuracy 


Individual-Level  Results 


Team  Results  (Time) 


E-E-E-E  (A)  — ■—  H-H-H-H  (D)  — ▲ —  E-E-H-H  (C)  H-H-E-E  (B) 


Team  Results:  Accuracy 


-------  E-E-E-E  (A)  ------  H-H-H-H  (D)  -- -a- -  -  E-E-H-H  (C)  -  - -x- -  -  H-H-E-E  (B) 


Implications  for  Theory 


□  Organizational  Forms 

■  We  can  think  of  Edge  and  Hierarchy  forms  as 
organizational  information  processing  structures  (i.e, 
March  and  Simon,  1950s) 

□  What  is  the  relationship  between  information  processing 
structures  and  an  organization's  capacity  for 
transferring  and  making  use  of  knowledge ? 

□  What  is  the  relationship  between  information  processing 
structures  and  volatility  when  organizations  learn? 

□  Organizational  Change 

■  Why  is  change  from  Edge  to  Hierarchy  detrimental  to 
performance? 

■  Why  is  change  from  Hierarchy  to  Edge  not 
detrimental  to  performance? 


Implications  for  Policy  and  Practice 

□  Example:  Hurricane  Katrina  Disaster  Relief 
Operations 

■  If  objective  performance  is  critical,  participants 
working  on  complex,  interdependent  tasks  of  disaster 
relief  should  lean  toward  Edge-like  organizing 

■  Switching  disaster  relief  efforts  from  Edge-like  to 
Hierarchy-like  organizing  will  degrade  performance, 
and  the  organization  will  perhaps  then  recover 

□  Anecdotal  evidence  suggests,  however,  that  agents 
"lower"  in  the  Hierarchy  may  limit  or  "opt  out"  of 
participation 

□  Consistent  with  prior  work  when  considered  in  terms  of  mutual 
adjustment  as  a  primary  coordination  mechanism  (Reeves  and 
Turner  1972;  Argote  1982) 


Traditional  C2  structures  are  ineffective  for  complex  tasks 

requiring  significant  coordination 


Implications  for  Policy  and  Practice 


□  In  uncertain,  new,  novel  □  In  known,  stable  and 

and/or  dynamic  task  conventional  task 

environments...  environments  ... 

■  Countering  insurgents  ■  Managing  HQ  "ops"  & 

■  Creating  new  interagency  "watch"  centers 

and  coalition  partnerships  ■  Producing  an  Air  Tasking 

■  Building  capacity  and  Order 

operating  procedures  for  ■  Teaching  basic  soldiering 

new  mission  areas  (e.g.,  skills 

counterpiracy ,  protecting 
economic  exclusion  zones) 

□  ...  Knowledge  sharing  and  □  ...Knowledge  sharing  and 

transfer  is  critical  transfer  is  less  critical 

□  "Edge-like"  organizing  is  □  "Hierarchy-like"  organizing 

more  optimal  may  produce  efficiencies 


“ Edge-like ”  organizing  is  more  optimal 
for  novel  or  dynamic  situations 


Summary 


□  Different  task  environments  call  for 
different  organizing  principles 

□  Edge  and  Hierarchy  have  unique  strengths 
and  weaknesses 

□  Most  optimal  form  depends  upon 
stakeholder  goals  for  organization 

□  "Gut"  instincts  may  not  enact  sufficient 
organizational  redesign 

■  Experimentation  is  critical  for  determining 
optimal  ways  to  organize  in  various  task 
contexts 
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